Vendor Evaluation

/arckit.evaluate <task description>

# Create evaluation framework
/arckit.evaluate Create framework for project 001

# Score individual vendor
/arckit.evaluate Score Acme Payment Solutions

# Compare all vendors
/arckit.evaluate Compare all vendors for payment gateway project

## Evaluation Criteria (100 Points)

### 1. Technical Approach (35 points)

**Solution Design (15 points):**
- Architecture quality and scalability
- Alignment with architecture principles
- Innovation and best practices
- Risk identification and mitigation

**Technical Compliance (10 points):**
- Coverage of MUST requirements
- Approach to NFR requirements (performance, security, scalability)
- Integration strategy

**Technology Choices (10 points):**
- Appropriate technology stack
- Justification for choices
- Future-proofing and maintainability

### 2. Project Approach (20 points)

**Methodology (8 points):**
- Development methodology (Agile, Waterfall, hybrid)
- Sprint planning and iteration approach
- Stakeholder engagement strategy

**Risk Management (6 points):**
- Risk identification completeness
- Mitigation strategies
- Contingency planning

**Quality Assurance (6 points):**
- Testing strategy (unit, integration, E2E, performance, security)
- Test coverage targets
- QA process and tools

### 3. Team Qualifications (25 points)

**Relevant Experience (12 points):**
- Domain expertise (financial services, healthcare, etc.)
- Technology stack experience
- Similar project delivery

**Team Composition (8 points):**
- Appropriate roles (architects, senior/junior devs, QA, DevOps)
- Team size and availability
- Key personnel CVs

**Certifications (5 points):**
- Professional certifications (AWS Certified, PCI-DSS QSA, etc.)
- Security clearances (if required)
- Training and continuous learning

### 4. Company Experience (10 points)

**Similar Projects (5 points):**
- Number and quality of reference projects
- Relevance to current project
- Client testimonials

**Industry Expertise (3 points):**
- Years in industry
- Market reputation
- Awards and recognition

**Financial Stability (2 points):**
- Company age and growth
- Client retention rate
- Financial health indicators

### 5. Pricing (10 points)

**Cost Competitiveness (5 points):**
- Total cost vs market benchmark
- Value for money assessment

**Pricing Clarity (3 points):**
- Transparent breakdown
- No hidden costs
- Clear payment terms

**Commercial Terms (2 points):**
- Payment milestone alignment
- Warranty terms
- Exit provisions

# Vendor Evaluation Framework

**Project**: Payment Gateway Modernization (001)
**Document**: ARC-001-EVAL-v1.0
**Created**: 2026-03-04

## Mandatory Qualifications (Pass/Fail)

Vendors must meet ALL criteria below or be disqualified:

- ✅ **PCI-DSS Certification**: Current QSA certification or 5+ years Level 1 experience
- ✅ **Financial Services Experience**: Minimum 5 years in payment systems
- ✅ **UK References**: Minimum 2 UK financial services references
- ✅ **Cloud Certification**: AWS Certified Solutions Architect (Professional) or equivalent
- ✅ **Team Availability**: Minimum 4 FTE committed for 6-month duration
- ✅ **Proposal Completeness**: All sections completed, no TBD items

## Scoring Criteria (100 Points)

[Full scoring criteria from template above]

## Evaluation Process

**Evaluation Team:**
- Lead Architect (RACI: Accountable)
- Senior Engineer (RACI: Consulted)
- Product Owner (RACI: Consulted)
- Procurement Manager (RACI: Informed)

**Timeline:**
1. Proposal deadline: 2026-03-15
2. Mandatory qualification check: 2026-03-16
3. Individual scoring: 2026-03-17 to 2026-03-20
4. Consensus meeting: 2026-03-21
5. Top 2 shortlist interviews: 2026-03-22 to 2026-03-25
6. Final decision: 2026-03-26
7. Contract award: 2026-03-27

**Decision Criteria:**
- Minimum score: 70/100 to be considered
- Top 2 scoring vendors invited for interview (technical deep-dive)
- Final decision based on: score (70%) + interview (20%) + cost (10%)
- Tie-breaker: Company with strongest PCI-DSS experience

## Audit Trail Requirements

- All scores documented with justification
- Evaluation notes for each vendor
- Consensus meeting minutes
- Interview scoring sheets
- Final decision rationale

# Vendor Evaluation: Acme Payment Solutions

**Project**: Payment Gateway Modernization (001)
**Vendor**: Acme Payment Solutions
**Proposal Date**: 2026-03-14
**Evaluated By**: Lead Architect
**Evaluation Date**: 2026-03-18

## Executive Summary

**Overall Score**: 76/100
**Rank**: 2nd of 3 vendors
**Recommendation**: CONSIDER (shortlist for interview)

**Strengths:**
- Strong PCI-DSS expertise (5+ Level 1 projects)
- Good reference projects (3 UK banks)
- Competitive pricing (£420k vs £480k average)

**Weaknesses:**
- Limited AWS cloud-native experience (mostly on-prem)
- Aggressive timeline (20 weeks vs 24 week recommendation)
- Team availability concerns (2.5 FTE vs 4 FTE requested)

**Risks:**
- 🟠 **Medium Risk**: Cloud-native architecture inexperience may impact NFR-P-001 (99.99% uptime)
- 🔵 **Low Risk**: Timeline pressure may affect code quality

## Mandatory Qualifications

| Qualification | Status | Evidence |
|---------------|--------|----------|
| PCI-DSS Certification | ✅ PASS | QSA certification #12345, expires 2027-06-30 |
| Financial Services Experience | ✅ PASS | 8 years, 5+ payment projects |
| UK References | ✅ PASS | 3 references provided (Barclays, HSBC, Lloyds) |
| Cloud Certification | ⚠️ CONDITIONAL | AWS Solutions Architect (Associate) - not Professional |
| Team Availability | ❌ FAIL | 2.5 FTE offered vs 4 FTE required |
| Proposal Completeness | ✅ PASS | All sections complete |

**Qualification Result**: ❌ **CONDITIONAL PASS** (2 issues: cloud cert level, team size)

**Action Required**:
- Clarify if team can be scaled to 4 FTE
- Request evidence of AWS cloud-native project delivery

---

## Detailed Scoring

### 1. Technical Approach (28/35 points)

#### Solution Design (12/15 points)
- **Architecture Quality** (4/5): Solid microservices design, good PCI segmentation
  - ✅ Meets PRIN-005 (API-first design)
  - ✅ PCI network segmentation follows best practices
  - ⚠️ AWS architecture lacks cloud-native patterns (ECS proposed, no Lambda/Fargate consideration)
  
- **Scalability** (4/5): Horizontal scaling via ECS auto-scaling
  - ✅ Meets NFR-SC-001 (auto-scaling to 1000 TPS)
  - ⚠️ Caching strategy limited (Redis only, no CloudFront CDN)
  
- **Innovation** (4/5): Standard approach, no standout innovations
  - ✅ ML fraud detection (FR-007)
  - ❌ No mention of chaos engineering (NFR-R-003)

#### Technical Compliance (8/10 points)
- **MUST Requirements Coverage** (6/7): 14/15 MUST requirements addressed
  - ✅ BR-001 (Faster Payments): Covered with API integration
  - ✅ BR-002 (PCI-DSS): Strong compliance approach
  - ✅ BR-003 (Open Banking): OBIE v3.1 compliant
  - ❌ NFR-P-001 (99.99% uptime): Only commits to 99.95% (gap!)
  
- **NFR Approach** (2/3): Good security, weak performance evidence
  - ✅ NFR-S-001 to NFR-S-008 (Security): Excellent
  - ⚠️ NFR-P-002 (< 3s processing): No load test evidence provided

#### Technology Choices (8/10 points)
- **Stack Appropriateness** (5/5): Go, PostgreSQL, Redis - solid choices
  - ✅ Aligns with PRIN-012 (open source preference)
  - ✅ Go performance for high-throughput (NFR-P-003)
  
- **Justification** (3/5): Limited justification for AWS service choices
  - ✅ PostgreSQL vs DynamoDB justified (ACID for payments)
  - ⚠️ ECS vs Lambda/Fargate not justified
  - ❌ No mention of AWS Well-Architected Framework

---

### 2. Project Approach (16/20 points)

#### Methodology (6/8 points)
- Agile with 2-week sprints
- Good sprint planning approach
- ⚠️ Limited stakeholder engagement (weekly demo vs daily collaboration)

#### Risk Management (5/6 points)
- 12 risks identified (good coverage)
- Mitigation strategies documented
- ⚠️ No mention of PCI QSA early engagement risk

#### Quality Assurance (5/6 points)
- >80% test coverage target
- ✅ PCI penetration testing included
- ⚠️ No chaos engineering / disaster recovery testing

---

### 3. Team Qualifications (18/25 points)

#### Relevant Experience (8/12 points)
- **Domain Expertise** (4/6): Strong PCI-DSS, weak cloud-native
  - ✅ 5+ PCI Level 1 projects
  - ⚠️ 1 AWS project (mostly on-prem experience)
  
- **Technology Stack** (4/6): Go, PostgreSQL strong; AWS weak

#### Team Composition (7/8 points)
- **Roles** (4/4): Architect, 2 senior devs, 1 junior, QA, DevOps
- **Availability** (3/4): 2.5 FTE vs 4 FTE requested (gap!)

#### Certifications (3/5 points)
- **PCI-DSS QSA**: ✅ Yes
- **AWS Certified**: ⚠️ Associate (not Professional)
- **Security Clearance**: N/A (not required)

---

### 4. Company Experience (8/10 points)

#### Similar Projects (4/5 points)
- 3 strong UK bank references
- Payment gateway modernization experience
- ⚠️ References are on-prem, not cloud

#### Industry Expertise (3/3 points)
- 8 years in payment systems
- Good reputation in UK FinTech
- ISO 9001 quality certification

#### Financial Stability (1/2 points)
- 8-year-old company (✅ stable)
- ⚠️ No financials provided (requested)

---

### 5. Pricing (6/10 points)

#### Cost Competitiveness (3/5 points)
- **Total Cost**: £420k (vs £480k average, £540k high)
- **Benchmark**: 12% below average (good value)
- ⚠️ Lowest bid (may indicate underestimation)

#### Pricing Clarity (2/3 points)
- Transparent breakdown by phase
- ⚠️ "Cloud infrastructure costs" TBD (hidden cost risk)

#### Commercial Terms (1/2 points)
- Milestone payments aligned
- ⚠️ Only 30-day warranty (vs 90-day requested)

---

## Risk Assessment

| Risk Area | Risk Level | Mitigation |
|-----------|------------|------------|
| **Cloud-native delivery** | 🟠 MEDIUM | Request AWS reference project proof |
| **Team capacity** | 🟠 MEDIUM | Clarify if team can scale to 4 FTE |
| **Uptime SLA gap** | 🟠 MEDIUM | Negotiate 99.99% or accept 99.95% |
| **Hidden cloud costs** | 🔵 LOW | Request fixed-price cloud infrastructure |
| **Aggressive timeline** | 🔵 LOW | Add 2-week contingency buffer |

**Overall Risk**: 🟠 **MEDIUM** (proceed with caution, clarify gaps)

---

## Recommendation

**CONSIDER** - Shortlist for technical interview

**Rationale**:
- ✅ Strong PCI-DSS expertise (critical for BR-002)
- ✅ Competitive pricing (12% below average)
- ✅ Good UK FinTech references
- ⚠️ Cloud-native experience gap needs validation
- ⚠️ Team capacity issue (2.5 vs 4 FTE)
- ⚠️ Uptime SLA gap (99.95% vs 99.99%)

**Interview Focus Areas**:
1. AWS cloud-native architecture deep-dive (request examples)
2. Team scaling plan to 4 FTE
3. Uptime SLA negotiation (99.99% commitment)
4. Load testing approach and evidence
5. Cloud cost estimation methodology

**Contract Negotiation Points** (if selected):
- Increase warranty to 90 days
- Fixed-price cloud infrastructure (no TBD costs)
- Commit to 99.99% uptime SLA with financial backing
- Minimum 4 FTE availability
- AWS Well-Architected review at HLD stage

# Vendor Comparison: Payment Gateway Modernization

**Project**: 001 - Payment Gateway Modernization
**Document**: ARC-001-VEND-v1.0
**Vendors Evaluated**: 3
**Date**: 2026-03-20

## Executive Summary

**Recommendation**: **BestPay Solutions** (82/100)

**Shortlist for Interview**: BestPay Solutions (82), Acme Payment Solutions (76)

**Eliminated**: CloudPayments Inc (71) - below 75-point threshold

---

## Overall Scores

| Rank | Vendor | Total Score | Technical | Project | Team | Experience | Pricing | Recommendation |
|------|--------|-------------|-----------|---------|------|------------|---------|----------------|
| 🥇 1st | **BestPay Solutions** | **82/100** | 33/35 | 18/20 | 22/25 | 9/10 | 0/10 | ✅ **RECOMMEND** |
| 🥈 2nd | Acme Payment Solutions | 76/100 | 28/35 | 16/20 | 18/25 | 8/10 | 6/10 | ⚠️ **CONSIDER** |
| 🥉 3rd | CloudPayments Inc | 71/100 | 24/35 | 14/20 | 20/25 | 7/10 | 6/10 | ❌ **NOT RECOMMENDED** |

---

## Detailed Comparison

### Technical Approach (35 points max)

| Criterion | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| Solution Design (15) | 14 | 12 | 10 |
| Technical Compliance (10) | 10 | 8 | 7 |
| Technology Choices (10) | 9 | 8 | 7 |
| **Subtotal** | **33** | **28** | **24** |

**Winner**: **BestPay Solutions**
- Best cloud-native architecture (Lambda, Fargate, serverless)
- 15/15 MUST requirements covered (Acme: 14/15, Cloud: 13/15)
- AWS Well-Architected Framework applied

**Acme Strengths**:
- Strong PCI segmentation
- Good microservices design

**Acme Weaknesses**:
- NFR-P-001 gap (99.95% vs 99.99% uptime)
- ECS-only (not cloud-native)

**CloudPayments Weaknesses**:
- Monolithic architecture (not microservices)
- 2 MUST requirements not addressed

---

### Team Qualifications (25 points max)

| Criterion | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| Relevant Experience (12) | 10 | 8 | 11 |
| Team Composition (8) | 8 | 7 | 6 |
| Certifications (5) | 4 | 3 | 3 |
| **Subtotal** | **22** | **18** | **20** |

**Winner**: **BestPay Solutions**
- AWS Certified Solutions Architect (Professional) ✅
- 4 FTE committed ✅
- 3 AWS cloud-native payment projects

**Acme**: Strong PCI-DSS, weak cloud (Associate cert, 2.5 FTE)

**CloudPayments**: Good experience, small team (3 FTE, availability concerns)

---

### Pricing (10 points max)

| Vendor | Total Cost | Points | Notes |
|--------|------------|--------|-------|
| Acme Payment Solutions | £420,000 | 6/10 | Lowest bid (12% below avg), TBD cloud costs |
| CloudPayments Inc | £480,000 | 6/10 | Average price, transparent |
| **BestPay Solutions** | £540,000 | 0/10 | **Highest bid (12% above avg)**, but best value |

**BestPay Pricing Justification**:
- 28% more than Acme (£540k vs £420k)
- BUT: Includes fixed-price cloud infrastructure (Acme = TBD)
- 99.99% uptime SLA (Acme = 99.95%)
- 4 FTE (Acme = 2.5 FTE)
- 90-day warranty (Acme = 30 days)
- **Adjusted TCO**: BestPay £540k vs Acme £480k (with cloud costs + team scaling)

---

## Strengths Comparison

### BestPay Solutions

- ✅ **Best technical approach**: Cloud-native, serverless, AWS Well-Architected
- ✅ **Best cloud experience**: 3 AWS payment projects, Professional cert
- ✅ **Best compliance**: Meets all 15 MUST requirements + 99.99% SLA
- ✅ **Best team**: 4 FTE, AWS expertise, good availability
- ✅ **Best commercial terms**: 90-day warranty, fixed cloud costs
- ❌ **Highest price**: £540k (but best value when adjusted for TCO)

### Acme Payment Solutions

- ✅ **Best PCI-DSS expertise**: QSA cert, 5+ Level 1 projects
- ✅ **Best pricing**: £420k (lowest bid)
- ✅ **Good UK references**: 3 UK banks (Barclays, HSBC, Lloyds)
- ⚠️ **Weak cloud-native**: ECS-only, Associate cert, 1 AWS project
- ⚠️ **Team capacity gap**: 2.5 FTE vs 4 FTE required
- ⚠️ **SLA gap**: 99.95% vs 99.99% uptime

### CloudPayments Inc

- ✅ **Good domain experience**: 6 payment projects
- ✅ **Transparent pricing**: £480k, clear breakdown
- ❌ **Weak architecture**: Monolithic, not microservices
- ❌ **Below threshold**: 71/100 (< 75 required)
- ❌ **2 MUST requirements**: Not addressed (BR-003, NFR-P-001)

---

## Risk Comparison

| Risk Area | BestPay | Acme | CloudPayments |
|-----------|---------|------|---------------|
| **Technical Delivery** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **Cloud-Native** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **PCI-DSS Compliance** | 🟢 LOW | 🟢 LOW | 🟠 MEDIUM |
| **Team Capacity** | 🟢 LOW | 🟠 MEDIUM | 🟠 MEDIUM |
| **Uptime SLA** | 🟢 LOW | 🟠 MEDIUM | 🔴 HIGH |
| **Cost Overrun** | 🟢 LOW | 🟠 MEDIUM | 🟢 LOW |
| **Overall Risk** | 🟢 **LOW** | 🟠 **MEDIUM** | 🔴 **HIGH** |

---

## Final Recommendation

### 🥇 Selected Vendor: **BestPay Solutions**

**Score**: 82/100 (highest)

**Decision Rationale**:
1. **Best Technical Fit**: Only vendor meeting all 15 MUST requirements with 99.99% SLA
2. **Lowest Technical Risk**: Proven AWS cloud-native expertise (3 payment projects)
3. **Best Long-Term Value**: £540k includes fixed cloud costs, 90-day warranty, 4 FTE
4. **Principle Alignment**: Strongest alignment with PRIN-001 (cloud-first) and PRIN-005 (API-first)
5. **Quality Over Cost**: 28% price premium justified by risk reduction and compliance

**Contract Negotiation**:
- Accept £540k pricing (justified by TCO analysis)
- Confirm 99.99% SLA with financial backing (SLA credits)
- Lock in 4 FTE availability (contractual commitment)
- Include AWS Well-Architected review at HLD stage

### 🥈 Runner-Up: Acme Payment Solutions

**Fallback Option If**: BestPay negotiations fail or budget constraints

**Required Clarifications**:
- Scale team to 4 FTE (increase £420k → £480k estimate)
- Commit to 99.99% uptime SLA
- Provide AWS cloud-native project evidence
- Fix cloud infrastructure costs (no TBD)

**Adjusted Score** (if gaps addressed): 81/100 (competitive)

### ❌ Not Recommended: CloudPayments Inc

**Reasons**:
- Below 75-point threshold (71/100)
- 2 MUST requirements not addressed (disqualifying gaps)
- High technical risk (monolithic architecture)

---

## Audit Trail

**Evaluation Panel**:
- Lead Architect (scoring + recommendation)
- Senior Engineer (technical review)
- Product Owner (business fit)
- Procurement Manager (commercial review)

**Consensus Meeting**: 2026-03-21, 14:00-16:00 GMT

**Attendees**: [List]

**Decision**: Unanimous recommendation for BestPay Solutions

**Dissenting Views**: None

**Interview Outcome** (if conducted):
- BestPay: Excellent technical deep-dive, AWS architecture validated
- Acme: Good PCI-DSS discussion, cloud-native concerns confirmed

**Final Approval**: [Pending] - Architecture Board, 2026-03-26