amfora.core.statistics

amfora.core.statistics#

Cross-sherd statistical analysis: PCA, clustering, hierarchical dendrograms, group comparisons. Operates on the DataFrame produced by amfora.full_analysis.

Source: src/amfora/core/statistics.py.

Convenience entry points#

amfora.core.statistics.quick_ceramic_analysis(data: DataFrame | dict, scaling_method: str = 'standard') → dict[source]#

One-liner convenience wrapper: prepare data and run generate_report().

Parameters:

  • data (pd.DataFrame or dict) – Ceramic analysis data (e.g. output of full_analysis()).

  • scaling_method (str, default 'standard') – Scaling method passed to prepare_data() — see that method for options ('standard', 'robust', 'none').

Returns:

Complete analysis report (same as generate_report()).

Return type:

dict

amfora.core.statistics.compare_assemblages(data: DataFrame, group_column: str, analysis_type: str = 'comprehensive') → dict[source]#

Compare ceramic assemblages between different archaeological contexts.

Note

Not yet implemented — raises NotImplementedError.

Parameters:

  • data (pd.DataFrame) – Ceramic data with group identifiers

  • group_column (str) – Column containing group/context identifiers

  • analysis_type (str, default 'comprehensive') – Type of analysis: 'basic', 'comprehensive', or 'advanced'

Returns:

Assemblage comparison results

Return type:

dict

Classes#

class amfora.core.statistics.CeramicStatisticalAnalyzer[source]#

Bases: object

Comprehensive statistical analysis class for ceramic fabric data.

This class provides all the statistical methods used in the streamlit application in a standalone, reusable format suitable for batch analysis and research.

Typical workflow:

analyzer = CeramicStatisticalAnalyzer()
analyzer.prepare_data(df, scaling_method='standard')

pca_results = analyzer.perform_pca(variance_threshold=0.90)
cluster_results = analyzer.perform_clustering(method='hierarchical')
corr_results = analyzer.correlation_analysis(method='pearson')

# Or run everything at once:
report = analyzer.generate_report()

Use CeramicVisualization to plot results (biplots, dendrograms, etc.).

assemblage_comparison(group_column: str, comparison_type: str = 'all') → dict[source]#

Compare ceramic assemblages between different groups.

Note

Not yet implemented — raises NotImplementedError.

Parameters:

  • group_column (str) – Column name containing group identifiers

  • comparison_type (str, default 'all') – Type of comparison: 'all', 'pairwise', or 'one_vs_rest'

Returns:

Comprehensive assemblage comparison results

Return type:

dict

correlation_analysis(method: str = 'pearson', significance_level: float = 0.05) → dict[source]#

Perform pairwise correlation analysis on ceramic features.

Computes the full correlation matrix, tests each pair for statistical significance, and returns significant pairs sorted by absolute strength.

Parameters:

  • method (str, default 'pearson') –

    Correlation coefficient to compute:

    • 'pearson' — linear correlation (parametric). Assumes approximately normal distributions.

    • 'spearman' — rank correlation (non-parametric). Robust to non-linearity and outliers; recommended when data are ordinal or heavily skewed.

    • 'kendall' — Kendall’s tau (non-parametric). More robust than Spearman for small sample sizes but slower to compute.

  • significance_level (float, default 0.05) – Alpha threshold for identifying significant correlations. Pairs with p < significance_level are included in the significant_correlations list.

Returns:

  • 'correlation_matrix' — DataFrame (features x features).

  • 'p_values' — DataFrame of p-values for each pair.

  • 'significant_correlations' — list of dicts sorted by |correlation|, each with feature1, feature2, correlation, p_value, and strength (Strong/Moderate/Weak/Very weak).

  • 'method', 'significance_level' — echo of inputs.

Return type:

dict

generate_report(include_plots: bool = True) → dict[source]#

Run PCA, hierarchical clustering, and correlation analysis in one call and bundle results into a single report dict.

Each analysis is run with default parameters (see perform_pca, perform_clustering, correlation_analysis for details). If any individual analysis fails, its entry will contain {'error': '<message>'} rather than raising.

Parameters:

include_plots (bool, default True) – Whether to include visualization plots in the report (currently unused — reserved for future HTML export).

Returns:

  • 'data_summary' — sample/feature counts and names.

  • 'pca_analysis' — output of perform_pca().

  • 'cluster_analysis' — output of perform_clustering().

  • 'correlation_analysis' — output of correlation_analysis().

  • 'archaeological_interpretation' — auto-generated summary of main findings and technological insights.

Return type:

dict

perform_clustering(method: str = 'hierarchical', n_clusters: int | None = None, linkage_method: str = 'ward', distance_metric: str = 'euclidean') → dict[source]#

Perform clustering analysis on ceramic data.

Requires prepare_data() to have been called first. When n_clusters is None, the optimal count is estimated automatically (elbow method for hierarchical, silhouette scan for k-means).

Parameters:

  • method (str, default 'hierarchical') –

    Clustering algorithm to use:

    • 'hierarchical' — agglomerative hierarchical clustering. Produces a linkage matrix suitable for dendrogram plotting via CeramicVisualization.plot_dendrogram().

    • 'kmeans' — K-means partitioning. Good when the number of groups is known or roughly estimated. Does not produce a dendrogram.

    • 'dbscan' — density-based clustering. Does not require n_clusters; discovers clusters of arbitrary shape and labels outliers as noise (cluster label -1). Parameters eps and min_samples are chosen automatically.

  • n_clusters (int, optional) – Number of clusters for hierarchical and k-means. Ignored by DBSCAN. If None (default), determined automatically.

  • linkage_method (str, default 'ward') –

    Linkage criterion for hierarchical clustering (ignored by k-means/DBSCAN). Passed to scipy.cluster.hierarchy.linkage():

    • 'ward' — minimizes within-cluster variance (requires Euclidean distance). Generally best for balanced clusters.

    • 'complete' — maximum inter-cluster distance. Tends to produce compact, equally-sized clusters.

    • 'average' — mean inter-cluster distance (UPGMA).

    • 'single' — minimum inter-cluster distance. Susceptible to chaining; useful for detecting elongated clusters.

    • 'centroid', 'median', 'weighted' — less common alternatives (see scipy docs).

  • distance_metric (str, default 'euclidean') –

    Distance metric for hierarchical clustering (ignored by k-means/DBSCAN). Any metric accepted by scipy.spatial.distance.pdist() is valid, including:

    • 'euclidean' — standard L2 distance. Required when linkage_method is 'ward'.

    • 'mahalanobis' — accounts for feature correlations and unequal variances. The inverse covariance matrix is computed automatically from the data. Well-suited for compositional / provenance studies. Falls back to pseudo-inverse with a warning if the covariance matrix is singular (e.g. when n < number of features). Incompatible with 'ward' linkage.

    • 'correlation' — 1 - Pearson r. Useful when the shape of the feature profile matters more than magnitude.

    • 'cosine', 'cityblock' (Manhattan), 'chebyshev', etc.

Returns:

Clustering results. Keys common to all methods:

  • 'method' — the method string used.

  • 'cluster_labels' — integer array of cluster assignments.

  • 'n_clusters' — number of clusters found.

  • 'silhouette_score' — mean silhouette coefficient (−1 to 1; higher is better). -1 if only one cluster.

  • 'calinski_harabasz_score' — Calinski-Harabasz index.

  • 'cophenetic_correlation' — cophenetic correlation coefficient (0 to 1; higher means the dendrogram faithfully represents pairwise distances). Only set for hierarchical clustering; None for k-means and DBSCAN.

  • 'cluster_summary' — per-cluster size and feature means.

  • 'sample_names' — list of sample labels (from filenames).

Additional keys by method:

  • hierarchical: 'linkage_matrix', 'linkage_method', 'distance_metric'.

  • kmeans: 'cluster_centers', 'inertia'.

  • dbscan: 'eps', 'min_samples', 'n_noise'.

Return type:

dict

perform_pca(n_components: int | None = None, variance_threshold: float = 0.95) → dict[source]#

Perform Principal Component Analysis on ceramic data.

Requires prepare_data() to have been called first. When n_components is not set, the number of components is chosen automatically as the fewest that cumulatively explain at least variance_threshold of the total variance (minimum 2).

Parameters:

  • n_components (int, optional) – Exact number of principal components to retain. If None (default), determined automatically from variance_threshold.

  • variance_threshold (float, default 0.95) – Cumulative explained-variance ratio at which to stop adding components. Only used when n_components is None. E.g. 0.90 keeps enough components to explain 90 % of variance — useful for reducing dimensionality while retaining most information.

Returns:

PCA results with keys:

  • 'scores' — DataFrame of sample scores (PC1, PC2, …) with a Sample column (from filenames).

  • 'loadings' — DataFrame of feature loadings per component.

  • 'explained_variance' — array of per-component variance ratios.

  • 'cumulative_variance' — cumulative sum of the above.

  • 'interpretations' — auto-generated archaeological interpretation of each component based on top loadings.

  • 'model' — fitted sklearn.decomposition.PCA object.

  • 'n_components' — number of components retained.

Return type:

dict

prepare_data(data: DataFrame | dict, exclude_columns: list[str] = None, include_only: list[str] = None, scaling_method: str = 'standard') → DataFrame[source]#

Prepare and preprocess ceramic analysis data for statistical analysis.

Must be called before any analysis method (perform_pca, perform_clustering, correlation_analysis, generate_report).

Non-numeric columns and constant columns (zero variance) are automatically removed. If the DataFrame contains a filename column, those values are stored as sample labels and used in PCA score tables and dendrogram leaf labels.

Parameters:

  • data (pd.DataFrame or dict) – Raw ceramic analysis data. If a dict, it is converted to a DataFrame first. Each row is one sherd/sample; columns are numeric measurement features (e.g. from full_analysis()).

  • exclude_columns (list of str, optional) – Column names to drop before analysis. Useful for removing metadata columns that survived automatic filtering, e.g. ['notes', 'context'].

  • include_only (list of str, optional) – If provided, only these columns are kept (after numeric filtering). Takes precedence — exclude_columns is applied first, then include_only filters the remainder.

  • scaling_method (str, default 'standard') –

    How to scale features before multivariate analysis:

    • 'standard' — zero-mean, unit-variance (scikit-learn StandardScaler). Best general-purpose choice.

    • 'robust' — median-centered, IQR-scaled (RobustScaler). Use when data contains outliers.

    • 'none' — no scaling. Only appropriate when all features share the same units and comparable ranges.

Returns:

Preprocessed, scaled data stored internally (also accessible as self.scaled_data).

Return type:

pd.DataFrame

Notes

Feature selection tips — AMFOrA full_analysis() can produce ~80+ columns per sherd, many of which are redundant (e.g. blob_mean_diameter_mm and blob_mean_area_mm2 measure essentially the same thing). Trimming redundant features improves clustering stability and is required for distance metrics like Mahalanobis (which need more samples than features for a well-conditioned covariance matrix).

Strategies for reducing feature count:

  1. Drop one from each highly-correlated pair. Run correlation_analysis() first and inspect significant_correlations — pairs with |r| > 0.9 are near-duplicates. Keep whichever is more interpretable.

  2. Pick one detection method. Blob and contour columns (blob_* vs contour_*) measure the same properties with different algorithms. Use include_only with one prefix, or exclude_columns to drop the other.

  3. Separate concerns. Color features (*_color_l/a/b), size features (*_diameter_*, *_area_*), and orientation features (*_orientation_*) capture different aspects of the fabric. For a focused analysis, include only the relevant group.

  4. Exclude summary statistics that overlap. Mean and median size, or count and density, often carry the same information. Keep one representative per concept.

class amfora.core.statistics.CeramicVisualization[source]#

Bases: object

Static visualization methods for ceramic statistical analysis results.

All methods are @staticmethod — no instantiation required:

from core.statistics import CeramicVisualization as viz

fig = viz.plot_pca_biplot(pca_results)
fig = viz.plot_dendrogram(cluster_results)
fig = viz.plot_correlation_heatmap(corr_results)

Every method returns a plotly.graph_objects.Figure that can be displayed with fig.show() or saved with fig.write_image().

static plot_archaeological_summary(report: dict) → Figure[source]#

Create a 2x2 dashboard summarizing PCA variance, cluster quality, top correlations, and sample distribution.

Parameters:

report (dict) – Output of CeramicStatisticalAnalyzer.generate_report().

Returns:

Four-panel summary figure.

Return type:

plotly.graph_objects.Figure

static plot_cluster_comparison(cluster_results: dict, feature_data: DataFrame, features_to_plot: list[str] = None) → Figure[source]#

Create box-plot grid comparing feature distributions across clusters.

Parameters:

  • cluster_results (dict) – Output of CeramicStatisticalAnalyzer.perform_clustering() (any method).

  • feature_data (pd.DataFrame) – The (unscaled) feature data — typically analyzer.scaled_data or the original DataFrame subset.

  • features_to_plot (list of str, optional) – Column names to include. If None, the first 4 columns of feature_data are used.

Returns:

Grid of box plots, one per feature, colored by cluster.

Return type:

plotly.graph_objects.Figure

static plot_correlation_heatmap(correlation_results: dict) → Figure[source]#

Create interactive correlation-matrix heatmap.

Parameters:

correlation_results (dict) – Output of CeramicStatisticalAnalyzer.correlation_analysis().

Returns:

Heatmap colored by correlation coefficient (−1 to +1).

Return type:

plotly.graph_objects.Figure

static plot_dendrogram(cluster_results: dict, orientation: str = 'bottom', max_labels: int = 20, sample_names: list = None) → Figure[source]#

Create hierarchical clustering dendrogram.

Parameters:

  • cluster_results (dict) – Results from hierarchical clustering

  • orientation (str, default 'bottom') – Dendrogram orientation

  • max_labels (int, default 20) – Maximum number of labels to show

  • sample_names (list, optional) – Labels for leaf nodes. If None, uses cluster_results[‘sample_names’] if available, otherwise falls back to numeric indices.

Returns:

Interactive dendrogram

Return type:

plotly.graph_objects.Figure

static plot_pca_3d(pca_results: dict, pc_x: str = 'PC1', pc_y: str = 'PC2', pc_z: str = 'PC3', cluster_labels: ndarray | None = None) → Figure[source]#

Create interactive 3D PCA scatter plot.

Parameters:

  • pca_results (dict) – Output of CeramicStatisticalAnalyzer.perform_pca().

  • pc_x (str, default 'PC1' / 'PC2' / 'PC3') – Principal components mapped to the x / y / z axes.

  • pc_y (str, default 'PC1' / 'PC2' / 'PC3') – Principal components mapped to the x / y / z axes.

  • pc_z (str, default 'PC1' / 'PC2' / 'PC3') – Principal components mapped to the x / y / z axes.

  • cluster_labels (array-like of int, optional) – Cluster assignment per sample (e.g. from perform_clustering()['cluster_labels']). When provided, points are color-coded by cluster; noise samples (label -1 from DBSCAN) appear as “Noise”.

Returns:

Interactive 3D scatter (rotate/zoom with mouse).

Return type:

plotly.graph_objects.Figure

static plot_pca_biplot(pca_results: dict, pc_x: str = 'PC1', pc_y: str = 'PC2', show_loadings: bool = True, max_arrows: int = 10) → Figure[source]#

Create PCA biplot with sample scores and feature-loading arrows.

Parameters:

  • pca_results (dict) – Output of CeramicStatisticalAnalyzer.perform_pca().

  • pc_x (str, default 'PC1' / 'PC2') – Which principal components to plot on the x/y axes. Use 'PC3', 'PC4', etc. to explore higher components.

  • pc_y (str, default 'PC1' / 'PC2') – Which principal components to plot on the x/y axes. Use 'PC3', 'PC4', etc. to explore higher components.

  • show_loadings (bool, default True) – Overlay feature-loading vectors as arrows. Arrows point in the direction each feature contributes to the two plotted components; longer arrows indicate stronger influence.

  • max_arrows (int, default 10) – Cap on the number of loading arrows shown (top features by combined loading magnitude). Reduces clutter in high- dimensional datasets.

Returns:

Interactive PCA biplot.

Return type:

plotly.graph_objects.Figure

static plot_scree_plot(pca_results: dict) → Figure[source]#

Create scree plot showing per-component explained variance.

Displays both individual (bars) and cumulative (line) variance ratios, helping determine how many components to retain.

Parameters:

pca_results (dict) – Output of CeramicStatisticalAnalyzer.perform_pca().

Returns:

Scree / elbow plot.

Return type:

plotly.graph_objects.Figure