Dimensionality reduction transforms high-dimensional data into lower dimensions while retaining important patterns. MLPP provides PCA and SVD for feature extraction and data compression.
Principal Component Analysis (PCA)
PCA identifies orthogonal directions of maximum variance and projects data onto principal components.
#include "Unsupervised Learning/Dimensionality Reduction/PCA.h"
#include <Eigen/Dense>
class PCA {
public:
PCA(int num_components);
void fit(const Eigen::MatrixXd& data);
Eigen::MatrixXd transform(const Eigen::MatrixXd& data);
Eigen::MatrixXd inverse_transform(const Eigen::MatrixXd& transformed_data);
Eigen::MatrixXd get_components();
Eigen::VectorXd get_mean();
};
Constructor
Parameters:
num_components - Number of principal components to retain
fit
void fit(const Eigen::MatrixXd& data)
data - Data matrix where rows are samples and columns are features
Process:
- Centers data by subtracting mean
- Computes covariance matrix
- Finds eigenvectors (principal components)
- Retains top
num_components by eigenvalue magnitude
Eigen::MatrixXd transform(const Eigen::MatrixXd& data)
data - Data matrix to transform (same feature dimension as training)
Returns:
Reduced representation with num_components features
Eigen::MatrixXd inverse_transform(const Eigen::MatrixXd& transformed_data)
transformed_data - Reduced data from transform()
Returns:
Approximate reconstruction in original feature space
Inverse transform is lossy - reconstructed data approximates the original by retaining only variance captured by selected components.
get_components
Eigen::MatrixXd get_components()
get_mean
Eigen::VectorXd get_mean()
Example usage
#include "Unsupervised Learning/Dimensionality Reduction/PCA.h"
#include <Eigen/Dense>
// Create high-dimensional data (100 samples, 50 features)
Eigen::MatrixXd data = Eigen::MatrixXd::Random(100, 50);
// Reduce to 10 dimensions
PCA pca(10);
pca.fit(data);
// Transform training data
Eigen::MatrixXd reduced = pca.transform(data);
// reduced is 100×10 matrix
// Transform new data
Eigen::MatrixXd new_data = Eigen::MatrixXd::Random(20, 50);
Eigen::MatrixXd new_reduced = pca.transform(new_data);
// Reconstruct approximation
Eigen::MatrixXd reconstructed = pca.inverse_transform(reduced);
// reconstructed is 100×50 matrix (approximate)
// Inspect components
Eigen::MatrixXd components = pca.get_components();
Eigen::VectorXd mean = pca.get_mean();
Singular Value Decomposition (SVD)
SVD factorizes data matrix into orthogonal components, providing optimal low-rank approximation.
#include "Unsupervised Learning/Dimensionality Reduction/SVD.h"
#include <Eigen/Dense>
class SVDReducer {
public:
void fit(const Eigen::MatrixXd& X);
Eigen::MatrixXd transform(std::size_t k) const;
Eigen::MatrixXd transform_new(const Eigen::MatrixXd& X_new, std::size_t k) const;
Eigen::MatrixXd reconstruct(const Eigen::MatrixXd& Z, std::size_t k) const;
const Eigen::MatrixXd& U() const;
const Eigen::VectorXd& S() const;
const Eigen::MatrixXd& V() const;
};
- U ∈ ℝ^(m×r) - Left singular vectors (sample space)
- Σ - Diagonal matrix with singular values σ₁ ≥ … ≥ σᵣ ≥ 0
- V ∈ ℝ^(n×r) - Right singular vectors (feature space)
- r = min(m, n)
Reduced representation: Z = U_k Σ_k ∈ ℝ^(m×k)
fit
void fit(const Eigen::MatrixXd& X)
X - Data matrix (rows = samples, columns = features)
Eigen::MatrixXd transform(std::size_t k) const
k - Target dimensionality (k < r)
Returns:
Z = U_k Σ_k, reduced representation of training data
Eigen::MatrixXd transform_new(const Eigen::MatrixXd& X_new, std::size_t k) const
X_new - New data matrix (same feature dimension)k - Target dimensionality
Returns:
X_new V_k, projection of new samples
reconstruct
Eigen::MatrixXd reconstruct(const Eigen::MatrixXd& Z, std::size_t k) const
Z - Reduced data from transform()k - Dimensionality used in reduction
Returns:
Rank-k approximation of original data (Eckart-Young-Mirsky theorem)
Accessors
const Eigen::MatrixXd& U() const // Left singular vectors
const Eigen::VectorXd& S() const // Singular values
const Eigen::MatrixXd& V() const // Right singular vectors
Example usage
#include "Unsupervised Learning/Dimensionality Reduction/SVD.h"
#include <Eigen/Dense>
// Create data matrix (150 samples, 80 features)
Eigen::MatrixXd X = Eigen::MatrixXd::Random(150, 80);
// Fit SVD
SVDReducer svd;
svd.fit(X);
// Reduce to 15 dimensions
Eigen::MatrixXd Z = svd.transform(15);
// Z is 150×15 matrix
// Transform new batch
Eigen::MatrixXd X_new = Eigen::MatrixXd::Random(30, 80);
Eigen::MatrixXd Z_new = svd.transform_new(X_new, 15);
// Z_new is 30×15 matrix
// Reconstruct approximation
Eigen::MatrixXd X_approx = svd.reconstruct(Z, 15);
// X_approx is 150×80 matrix (rank-15 approximation)
// Access singular components
Eigen::MatrixXd U = svd.U();
Eigen::VectorXd S = svd.S(); // Singular values
Eigen::MatrixXd V = svd.V();
SVD provides the optimal low-rank approximation under the Frobenius norm. The first k singular vectors capture the most variance.
PCA vs SVD
| Feature | PCA | SVD |
|---|
| Centering | Automatically centers data | User must center if needed |
| Use case | Feature extraction, variance analysis | Low-rank approximation, compression |
| Output | Principal components | Full decomposition (U, Σ, V) |
| New data | transform() for new samples | transform_new() for new samples |
| Flexibility | Fixed num_components | Choose k at transform time |
- You want automatic mean centering
- Focus is on variance-based feature extraction
- Standard workflow for preprocessing
Choose SVD when:
- You need full matrix decomposition
- Working with recommender systems or matrix completion
- Require flexibility in choosing dimensionality post-fit
- Reduce dimensions before clustering - Speeds up K-means and improves results
- Visualize with 2-3 components - Use PCA/SVD for exploratory data analysis
- Denoise signals - Truncate small singular values to filter noise
- Analyze variance - Inspect singular values to choose optimal k
- Apply to image compression and latent semantic analysis
- Use as preprocessing for supervised learning algorithms
- Combine with clustering for better feature spaces