FSDA TOOLBOX (Release 2023a available from April, 2023).
 |
FSDA TOOLBOX can be directly installed via Matlab Add-ons. |
|---|---|
 |
For Developers. FSDA TOOLBOX is now available in Github, where we host our current code release, old releases and docs. Please report issues and submit patches. |
Highlights of the last releases
FSDA follows typical MATLAB timetable in the sense that there are two releases per year. The first typically in May/June and the second around October/November.
We decided to label FSDA release with the most recent MATLAB version to which it is associated.
|
REGRESSION Now function LTSts also accepts missing values in the response GRAPHICS New option Varnames in aceplot. New option showBars in avasms New function avasmsplot which enables: to display the solutions found using function avasms (model selection using robust avas). Using option databrush it is possible to see after interactive brushing to see the details of each solution. New options colorBackground and typespm in spmplot. MULTIVARIATE ANALYSIS New output in pcaFS: orthogonal distance from the principal component subspace, and score distance. A new plot which shows the orthogonal distance vs the score distance is also autoamtically produced. New input in pcaFS: using options bsb or bdp it is possible to have a robust estimate of the covariance matrix. New sliders snd buttons in APP biplotAPP to show how the biplot changes varying breakdown point or subset size. HTML HELP DOCUMENTATION The new category "Probability distribution" has been created. This section contains the FSDA routines for correlation coefficient, inverse gamma, mixture of non central chi-square, Tweedie, Vervaat and Wallenius distribution DATASETS The description of the datasets (clustering, multivariate or regression) has been added inside the table. Now when a dataset in table format is loaded with DataSetName.Properties.Description it is possibile to see the description of the dataset. For example after: load geyser.mat the instruction geyser.Properties.Description enables to see the description. New datasets BigTradeEvents (section clsutering) and clothes33 (section multivariate) and leafpine (section regression) UTILITIES Function tabledisp in order to properly display a table or an array from the command line (for example with prespecified number of decimals) Function crosstab2datamatrix now produces in output also a table. json function signature definitions added to subfolder clustering which allow contextual help when calling FSDA functions GUIs New option inferential in GUIregress.m which shows the detailed calculations to compute t-statistics and confidence intervals. New option DiscreteData in GUImad.m New function GUIautocorr which shows the calculations to obtain the autocorrelation. |
|
DATASETS New datasets kidney, melody and wholesale, added to the section clustering datasets. Description added to datasets inside the table format UTILITIES STAT New functions Mlocation and Mlocsca added. Mlocation performs M estimate of location in univariate samples. Mlocsca computes simultaneously estimate of location and scale.
New functions corrpdf and
corrcdf to compute the pdf
and cdf of the sample correlation coefficient REGRESSION CLUSTERING tclustreg returns in output the value of the objective function for each subset after the concentration steps.
forecastTS improved to cope with autoregressive component. Decomposition of forecasts into the different underlying components when input option plots is equal to 2
LTStsVarSel.m new option firstTestLS
GUIs New function GUIcorr added to show the calculations to compute the correlation coefficient
|
|
GRAPHICS New option PlotGroup in scatterboxplot.m Improved mdpattern with syncronized axes. New options plots.ColorMapLabelRows and plots.ColorMapLabelRows in CorAnaplot in order to have a colorbar or row or column labels depending on a particular variable such as the corresponding mass or the communaility.
REGRESSION Added options rob, tyinitial, trapezoid, orderR2 and scail to avas. New function avasms, to automatically choose the best AVAS model. Added a series of options to avasms. A new graphical display to show the importance of the different options is shown in the output. DATASETS New datasets fish and Marketing_Data added to regression datasets. UTILITIES HELP Updated and improved function CreateFSDAtoolboxFile which automatizes all steps needed to create programmatically a toolbox (mltbx) file and github release New function setToolboxStartEnd, which can set release compatibility in ToolboxPackagingConfiguration.prj file.
CLUSTERING Improved version of ctlcurves with ordered solutions based on LR test New options startv1true1unitCentroid and priorSol added to tclust
GRAPHICS New options oneplot added to aceplot.m |
|
ROBUST CLUSTERING Likelihood ratio test inside ctlcurves.m and extended output which automatically detects a set of candidate solutions. GRAPHICS New function ctlcurvesplot.m to plot the output of the trimmed likelihood curves New function moonplot to show the output of correspondence analysis New function balloonplot to visualize large categorical data DATASETS New datasets oliveoil, wine and flea added to cluster analysis section. New datasets csdPerceptions and mobilephone added to the multivariate analysis section MULTIVARIATE ANALYSIS New function mcdeda which monitors the output of mcd for a sequence of values of breakdown point New option conflimMethodCramerV in function corrNominal to compute the confidence interval of Cramer V coefficient. New function mdpattern which finds and plots missing data patterns. REGRESSION New options inside AVAS to make it robust to the presence of outliers Sreg and MMreg with hyperbolic tangent estimator made faster. New options inside LTSts and simulateTS that allow a customized definition of the autoregressive component
UTILITIES Added weights in repDupValWithMean.m New function setdiffFS in case input arguments just contain positive integer values. |
|
MULTIVARIATE ANALYSIS New function pcaFS which enables to visualize both the percentage of variable explained by the different components, both the factor loadings and interactive dynmic biplot. New function biplotFS which enables to call the app biplotAPP which creates an interactive dynamic biplot. Improvements in functions spmplot (now input can also be a table) and CorAna (now input can also be a table when optional option datamatrix is true) CORRESPONDENCE ANALYSIS A new suite of functions for robust correspondence analysis. Functions mcdCorAna, FSCorAnaeda, FSCorAnaenvmmd and mahalCorAna, extend functions mcd, FSMeda, FSMenvmmd and mahalFS to the context of contingency tables. The plots produced by functions malindexplot and malfwdplot have been modified to incorporate the output of these new functions. GRAPHICS New function barVariableWidth which produces a bar plot with different widths and colors for each bar In functions mmdplot, mdrplot, mmdrsplot and mdrrsplot added the possibility of showing in output arguments the list of units brushed in each brushing operation Option tag added to FSR, FSRmdr, FSM, FSMmmd, FSRts, FSRB, FSRr and FSRH Option label added to malindexplot ROBUST CLUSTERING newoption warmup in tclustICgpcm new option commonslope in tclustreg and tclustregeda and tclustregIC DATASETS New dataset citiesItaly added to multivariate analysis section. REGRESSION New options for VIOM and MSOM contamination inside function simulateLM STATISTICAL UTILITIES Added a series of routines for the estimation of integrated and instantaneous variance of a diffusion process via Fourier analysis [M.E. Mancino, M.C. Recchioni, S. Sanfelici, ``Fourier-Malliavin Volatility Estimation. Theory and Practice'', 2017, Springer - New York] FE_int_vol, FE_int_vol_Fejer, FE_splot_vol, FE_spot_vol_FFT, CEVmodel, OptimalCuttingFrequency GUIs A new set of functions which enable to monitor all the necessary steps to be done to obtain the main indexes of descriptve statistics. More in detail: GUIconcentration, GUImad.html, GUIpowermean.html, GUIquantile.html, GUIskewness.html, GUIstd.html, GUItrimmean.html, GUIvar.html, GUIcov.html |
|
TRANFORMATIONS IN REGRESSION We have enriched the properties of the data transformations of the Yeo and Johnson (2000) for negative and positive responses, which we introduced in R2020a. More specifically, we intervened on the smoothness condition that the second derivative of zYJ(lambda) with respect to y be smooth at y = 0, along Atkinson et al (2019) and (2020), to allow two values of the transformations parameter: lambdaN for negative observations and lambdaP for non-negative ones. Now, function ScoreYJall computes: 1) a global t test associated with the constructed variable for lambda=lambdaP=lambdaN. 2) a t test for positive observations. 3) a t test for negative observations. 4) a F test for the joint presence of the two constructed variables described in points 2) and 3. 5) the F test based on the maximum liklihood estimate of lambdaP and lambdaNNew function ScoreYJall which computes the score tests described in points 1)-5) above. New function scoreYJmle which computes, in the case of extended Yeo and Johnson transformation, the likelihood ratio test of H0: lamabdaP=lamabdaP0 and lamabdaN=lambdaNeg0. Added option usefmin in funtion boxcoxR. This option uses the solver (fminsearch or fminunc) to find MLE of the two transformation parameters for extended Yeo and Johnson family (Atkinson et al. 2020). New function fanBIC which takes in input the output of FSRfan and using BIC and smoothness index enables to automatically choose in an effcieint and robust way, the best value of the transformation parameter. New function fanBICpn which enables to automatically choose the best values of the transformation parameters for positive and negative observations New function normYJpn which extends the companion functions normBoxCox and normYJ to the case of extended Yeo and Johnson transformation. TIME SERIES New function SETARX which implements Threshold autoregressive models with two regimes ROBUST CLUSTERING New tools for dealing with the 14 Gaussian parsimonious clustering models (GPCM). In function genSigmaGPCM new option pa.exactrestriction has been added. If pa.exactrestriction is true the covariance matrices are generated with the exact values of the restrictions specified in pa.cdet, pa.shw and pa.swb. In function MixSim optional input structure sph now can be called with field sph.exactrestriction In function tclust the
fourth input restrfactor can be a structure which can contain the
type of Gaussian Parsimonious Clustering Model - GPCM (restrfactor.pars),
the New functions tclustICgpcm, tclustICsolGPCM, tclustICplotGPCM, and carbikeplotGPCM which extend functions tclustIC, tclustICsol, tclustICplot and carbikeplot to the case of the 14 GPCM. GRAPHICS New plots waterfallchart
(which implements the waterfall chart (see
https://en.wikipedia.org/wiki/Waterfall_chart) and new function
funnelchart which implements the
funnel chart (see new function scatterboxplot (which creates scatter diagram with marginal boxplots). Improvment to functions Now spmplot accepts as input a table. In the case the names of the tables are automatically added at the marings. Similarly, in function corrNominal when option datamatrix is true it is possible to supply as first argument a table.
DATASETS New datasets balancesheets and facemasks added in the datasets regression section and datasets clustering section respectively |
|
ROBUST REGRESSION New set of routines for minimum density power divergence estimators (mdpd, mdpdR, mdpdReda, PDrho, PDpsi, PDwei, PDpsider, PDpsix, PDbdp, PDeff, PDc), discussed in https://www.mdpi.com/1099-4300/22/4/399. New function simulateLM to simulate linear regression data with prespecified value of R2, prespecified correlation among the explanatory variables and type of distribution. New function VIOM which computes weights estimates under a Variance-Inflation Outlier Model using MLE or Restricted MLE (REMLE). ROBUST CLUSTERING New function tclustregeda which enables to monitor the regression clustering classification for different levels of trimming. Improved function tclustregIC which enables to compute the BIC (and other information criteria) for different values of restriction factors and different number of groups, for classification or mixture likelihood and regression clustering. Modified function tclustICsol now accepts input from tclustregIC to show the yXplot of the best solutions. New function ctlcurves to select the appropriate number of groups in robust clustering. New function mdrrsplot which plots the random starts trajectories and enables to brush them. The companion function mmdrsplot referred to multivariate analysis has been improved. TRANFORMATIONS IN REGRESSION New function boxcoxR which computes the profile log Likelihood for a range of values of the transforamtion parameter (lambda) and computes the MLE of lambda in the supplied range. Supported families Box Cox, Yeo and Johnson and extended Yeo and Johnson (Atkinson et al. 2020). DOCUMENTATION Improved menu for the automatic installation of the FSDA html help files. UTILITIES New function exactcdf for finding the exact cdf of each element in a vector x with respect to the empirical distribution, represented by another vector. New functions twdpdf and twdrnd to compute the pdf of the Tweedie distribution and generate random numbers from it. |
|
This is the first release which is distributed from Mathworks marketplace and from github platform. TRANSFORMATION IN REGRESSION New function tBothSides which enables to transform both sides of a (nonlinear) regression model. New function boxcoxR which finds MLE of lambda in linear regression (and confidence interval) using Box Cox or Yeo and Johnson family. New function boxcoxR which finds MLE of lambda in linear regression (and confidence interval) using Box Cox or Yeo and Johnson family. ROBUST TIME SERIES ANALYSIS New functions LTStsVarSel which enables to perform variable selection in the robust time series model LTSts. In functions LTSts.m, simulateTS and forecastTS it is now possible to add an autoreressive component. UTILITIES New function existFS which checks whether a file exists and puts the answer in a cached persistent variable DOCUMENTATION Added file getting_started.mlx in subfolder doc of the main root of FSDA for packaging the FSDA toolbox. |
|
CLUSTER ANALYSIS Function tclustreg has been considerably enhanced. Now the function includes: (i) robust BIC, (ii) possibility of constraining the determinants of the covariance matrices of the explanatory variables, (iii) options for treating datasets with concentrated noise, making use of concentration steps appropriately modified using observation weighting and thinning methods. New function tclustregIC which (if present) uses the Parallel Computing toolbox to compute robust BIC for mixture and classification likeilhood for different values of k (number of groups) and different values of c (restriction factor for the variances of the residuals), for a prespecified level of trimming. New function for constraining the determinants restrdeter. This function has its own interest but is called in every concentration step of function tclust in case determinant restriction is needed. Routines for constraining the determinants (restrdeterGPCM), the shape matrices (restrshapeGPCM) and to impose common rotation matrices (common principal components) in presense of equal shape (cpcE.m) or varying shape (cpcV.m) and a general routine to impose constraints in the family of the 14 Gaussian Parsimonious Clustering Models (restrSigmaGPCM). Routine to generate data based on the 14 Gaussian Parsimonious Clustering Models (genSigmaGPCM). This routine can be called directly from function MixSim in order to generate each of the 14 Gaussian Parsimonious Clustering Models with a prespecified level of overlap (see option sph inside MixSim). Routine GowerIndex to compute matrix of similarity indexes using Gower metric. DATASETS New datasets added to the collection: animals, P12119085, P17049075, fondi_large, JohnDraper data, gasoline data, ms212. See pages datasets_reg and datasets_mult for a description of these datasets. GRAPHICS Possibility of brushing using rownames. Rownames also appear in the associated scatter plot matrix, both for regression and multivariate analysis: se new examples in resfwdplot and malfwdplot. New function aceplot to visualize the results of the output produced by functions ace and avas. Option RowNamesLabels has been added to add2spm and to add2yX to label the units. MULTIVARITATE Funciton FSMeda is now much faster; the original function FSMeda has been kept, renamed FSMedaeasy, because the algorithm is much easier to follow. REGRESSION New functions: (i) ace which implements the alternating conditional expectations algorithm to find the transformations of y and X that maximise the proportion of variation in y explained by X and (ii) avas which uses a (nonparametric) variance-stabilizing transformation for the response variable. New function smothr to smooth values imposing variour constraints (e.g. monotonicity, circularity,..). This function calls the supersmoother routine of Friedman. New function rlssmo to compute a running line smoother with global cross validation. New function supsmu to smooth scatterplots using Friedman's supersmoother algorithm. Function RobCov now includes the estimator covrobc (a corrected version of the covariance matrix of robust beta coefficients). A new motivating example shows a case why covrobc should be always used. UTILITIES New function repDupValWithMean that enable to replace values of y including non unique elements in vector x with local means. UTILITIES HELP Function publishFS is fourthly improved. This function automatically transforms structuerd .m files into MATLAB pure style files. In the HTML help files now the right click of the mouse (similarly to pure Mathworks pages) enables to execute, select or find help (F1 key) for all the versions of MATLAB starting from 2017a.
STATISTICAL UTITLIES New function genr8 to generate random numbers which are coherent across different software platforms. New function exactcdf to find exact cdf values of each element of an input vector x with respect to an empirical distribution. New function wthin which thins a uni/bi-dimensional dataset. New function ctsub which computes numerical integrarion from x(1) to z(i) of y=f(x) New functions (i) vervaatsim
(to simulate precisely from a Vervaat perpetuity
|
September 2018 Release 2018b
(1) New function qqplotFS that enables to create a qqplot of residuals with confidence bands (2) New function mtR which generates the same random numbers produced by R software with Mersenne Twister mt19937ar (3) New functions associated with Rocke biweght estimator. See for example RKrho, RKpsi, RKpsider, RKwei, RKbdp, RKeff. (4) Routines FSR, FSRmdr, FSRbsb extended to time series (see new functions FSRts, FSRtsmdr, FSRtsbsb and regressts) (5) New function verlessthanFS. It is a faster version of MATLAB function verlessthan. (6) New datasets added to the collection. (7) New routine publishBibliography to create in a automatic way the bibliography from the citations present inside the .m files. |
June 2018 Release 2018a
(1) New function tclusteda that helps choosing the best tclust model. It computes tclustfor different values of the trimming factor and produces plots that allow to find the optimal level of trimming. This function uses the parallel processing toolbox, if available. (2) Extension of the score test. New function ScoreYJpn that computes the score test for Yeo Johnson transformation separately for positive and negative observations. FSRfan now accepts the new option family "YJpn" and it is possible to monitor the score test for both positive and negative observations (output arguments out.Scorep and out.Scoren). (4) New functions for time series analysis. simulateTS simulates a time series with trend (up to third order), seasonality (constant or of varying amplitude) with a different number of harmonics and a level shift. forecastTS produces forecasts with confidence bands for a time series estimated with function LTSts. (4) CorAna has an improved display of results. New function CorAnaplot draws a rich Correspondence Analysis graph with different types of confidence ellipses for selected rows and columns. (5) New function verlessthanExt. It is a faster version of MATLAB function verlessthan. (6) Documentation of yXplot considerably improved. New options added (xlimx, ylimy, namey, nameX). (7) MixSimreg extended to account for multiple parameter distribution (betadistrib option) (8) histFS has a new optional argument (weights) for plotting a weighted histogram. (9) options labenv has been added to mmdrsplot. (10) option axesellipse added to ellipse (11) New output argument OldAndNewIndexes used in function UnitsSameCluster, to track the indexes permutations used to rearch a desired cluster labelling.
|
November 2017 Release 2017b.
| Major statistical release. Highlights:
FSDA has introduced two new categories of tools, one for (robust) time series analysis; another for analyzing categorical data and contingency tables. More precisely: (1) Function
LTSts extends LTS estimator to time series. A related new graphical plot associated to a time series,
wedgeplot, provides information on the presence of outliers and level shifts. (2) CorAna performs correspondence analysis; SparseTableTest computes independence test for large and sparse contingency tables; CressieRead computes the power divergence family of tests, to check the discrepancy/distance between observed and expected frequencies in a contingency table; rcontFS generates a random two-way table with given marginal totals; barnardtest computes the Barnard test, corrNominal measures strength of association between two unordered (nominal) categorical variables. Similarly for ordinal data with corrOrdinal. crosstab2datamatrix recreates the original data matrix X from contingency table N. This group of functions is complemented by file examples_categorical.m as in style of FSDA. The two categories of functions will be progressively enriched. Other new functions which are included are boxtest (test of equality of covariance matrices used for example in tkmeans), GYfilt (Gervini and Yohai, univariate outlier identifier), mmdrsplot (interactive plot of the trajectories of minimum Mahalanobis distances from different starting points), overlapmap to plot the ordered pairwise overlap values between components, dempk to perform a merging of components found by tkmeans, ncpci to compute a non centrality parameter confidence interval. Finally, spmplot has been enriched to superimpose ellipses, density and contour functions to data and extract single panels from the scatter matrix. |
May 2017 Release 2017a.
Major statistical release. Highlights:
|
October 2016 Release 2016b.
Major statistical release. Highlights:
|
June 2016 Release 2016a.
Major statistical release. Highlights:
Major highlights in the documentations and examples:
|
September 2015 Release 2015b.
|
Please, do not hesitate to contact us for any bug you might find and for any suggestion you might have!!! Thanks to all those who have already contacted us and have helped us to correct several bugs and improve the performance of the code.
Marco Riani, Andrea Cerioli and Aldo Corbellini (University of Parma)
Domenico Perrotta and Francesca Torti (EC, Joint Research Centre)
Patrizia Calcaterra, Andrea Cerasa, Emmanuele Sordini, Daniele Palermo (EC, Joint Research Centre)