🏗️ System Architecture & Common Knowledge¶

Essential knowledge for all team members

High-Level Overview¶

graph TB
    subgraph "API Layer"
        A[KONG API Gateway]
    end

    subgraph "Application Services"
        B[Appgain Server]
        C[Parse Server]
        D[Notify Service]
        E[Automator Engine]
        F[Admin Server]
        G[Email Platform - Mailu]
        H[AppBackend Server]
        I[Task Queue]
        J[Push Sender - FCM/APN]
        K[Shrinkit Shopify Backend]
    end

    subgraph "Data Layer"
        K[MongoDB Clusters]
        L[Central Redis]
        M[PostgreSQL - KONG Only]
    end

    subgraph "External Services"
        N[Web Command Runner]
        O[Email Warm-up Service]
    end

    A --> B
    A --> C
    A --> D
    A --> E
    A --> F
    A --> G
    A --> H
    A --> I
    A --> J
    A --> K

    B --> L
    C --> L
    D --> M
    E --> I
    F --> L
    G --> M
    H --> L
    H --> M
    K --> L

    D --> J
    E --> N
    G --> O

Infrastructure Components¶

Server Infrastructure¶

🖥️ Production Servers (OVH Cloud)

🗄️ Data Infrastructure

☁️ Cloud Infrastructure

📊 ERP Infrastructure

Quick Access Commands¶

# Core Production Services
ovh-appgain-server    # Main application server
ovh-parse-server      # Parse server and APIs
ovh-kong             # API Gateway
ovh-pushsender       # Push notification service
ovh-shopify-backend  # Shrinkit Shopify Backend

# Development & Staging
ovh-devops           # DevOps environment
ovh-preprod          # Pre-production testing

# Data & Analytics
ovh-growthmachine    # Growth analytics and ML
ovh-n8n              # Workflow automation and data integration

# Database Servers
ovh-mongo-master     # MongoDB primary
ovh-mongo-slave1     # MongoDB replica 1
ovh-mongo-slave2     # MongoDB replica 2

# Specialized Applications
ovh-ikhair           # Donation payment platform
ovh-ikhair-db        # Donation payment database
ovh-shrinkit         # URL shortening service
ovh-wasl             # Wasl platform
ovh-wasl-db          # Wasl database

# Cloud Infrastructure
OVH-efs              # OVH file storage
zafra-apps           # Azure applications
zafra-db             # Azure database
ihsan-app            # Azure healthcare app
ihsan-db             # Azure healthcare database

# ERP Systems
odoo-preprod         # Odoo testing environment
odoo-prod            # Odoo production system

Database Entities¶

User & Subscription Management¶

Users:
  # Authentication & Security
  - bcrypt encrypted passwords, email verification
  - Profiles: name, email, company, business_domain
  - KYC: revenue_range, country, mobile_number
  - Billing: stripe_customer_id, payment status
  - Types: self, enterprise, internal
  - Status: active, suspended, deleted

Subscriptions:
  # Plan & Billing Management
  - Plan management with renewal synchronization
  - Billing cycles: start_date, end_date
  - Usage tracking vs plan limits
  - Stripe integration for payments

Application Architecture¶

Apps (Suites):
  # Multi-tenant App Management
  - Multi-tenant: suit_name, subdomain, type
  - Configuration: app_icon, web_url, deep_links
  - Store Links: apple_store_id, google_play_id
  - Features: smartlink, parse, marketing_links
  - Status: active, disabled, production/staging

Plans:
  # Subscription Tiers & Limits
  - Tiers: Free, Starter, Pro, VAS
  - Limits: MarketingLinks, Push notifications, Storage
  - Usage types: gb, hit, push, event
  - Trial periods available

Marketing & Analytics¶

Smart_Links:
  # Dynamic Link Management
  - Dynamic routing: launch_page, platform targets
  - Attribution: FB, Google, Twitter, LinkedIn, TikTok
  - Conversion tracking: web_push, app_installed
  - Analytics: platform, browser, location data

Notification_Campaigns:
  # Multi-channel Messaging
  - Channels: appPush, webPush, SMS, email
  - Scheduling: immediate, scheduled, recurrent
  - Automation: autoMessage integration
  - Analytics: performance, conversion rates

Postman Collections Overview¶

🎯 Primary Production Workspaces - Appgain.io Workspace: 33 collections, 20 environments - iKhair Healthcare Platform: 11 collections, 10 environments - Outsourced Development: 45 collections, 15 environments - RetailGain Platform: 28 collections, 12 environments

📊 Collection Categories - Core Platform: Appgain Server, Notify, Automator, Auth Service - Integration: Shopify, Salla, Zid, Stripe, Google Ads - Data & Analytics: Data Engineering, Growth Machine, iKhair Data - Testing & Development: Testing Automation, Development APIs

Notify Service Architecture¶

📱 Multi-Channel Notification System - Purpose: Centralized notification service for all communication channels - Technology: Node.js microservice with channel-specific adapters - Architecture: Plugin-based architecture supporting multiple channels - Features: Push notifications, SMS, Email, WhatsApp, Web Push

🔧 Channel Components - Push Notifications: FCM (Android) and APNs (iOS) integration - SMS Gateway: Twilio, RouteMobile, and local SMS providers - Email Service: SMTP integration with Mailu and external providers - WhatsApp Business: Meta WhatsApp Business API integration - WhatsApp Lite Channel: Optimized high-volume messaging - Web Push: Browser-based push notifications

📊 Channel Capabilities

🔗 Integration Points - Appgain Server: User management and preferences - Parse Server: Mobile app notifications - Automator Engine: Automated campaign triggers - CRM Systems: Customer data and segmentation - Analytics: Delivery tracking and performance metrics

📈 Performance Metrics - Throughput: 10,000+ messages per minute - Delivery Rate: 95%+ successful delivery - Latency: < 5 seconds average delivery time - Uptime: 99.9% service availability

🔧 Configuration & Management - Channel Registration: Dynamic channel discovery and registration - Template Management: Pre-approved message templates - Rate Limiting: Channel-specific rate limiting and throttling - Retry Logic: Exponential backoff for failed deliveries - Webhook Support: Real-time delivery status updates

📚 Documentation - Notify API: API Documentation - WhatsApp Lite Channel: WhatsApp Lite Channel - Channel Setup: Channel Configuration Guide

Task Queue Architecture¶

🔄 Background Job Processing System - Purpose: Asynchronous task processing and job scheduling - Technology: Redis with Bull/BullMQ for Node.js applications - Architecture: Distributed job queue with priority and retry mechanisms - Features: Job scheduling, retry logic, progress tracking, job cancellation

🔧 Core Components - Job Producer: Services that create and queue jobs - Job Consumer: Workers that process queued jobs - Job Scheduler: Time-based job scheduling and recurring tasks - Job Monitor: Real-time job status and progress tracking - Job Cleanup: Automatic cleanup of completed and failed jobs

📊 Job Types & Priorities

🔗 Integration Points - Appgain Server: User data processing and analytics - Notify Service: Scheduled notifications and campaigns - Automator Engine: Workflow automation triggers - Email Platform: Bulk email processing - Analytics: Data processing and report generation

📈 Performance Metrics - Throughput: 1,000+ jobs per minute - Job Success Rate: 98%+ successful completion - Average Processing Time: < 5 minutes per job - Queue Depth: < 1,000 pending jobs

🔧 Configuration & Management - Queue Configuration: Redis connection and queue settings - Worker Scaling: Horizontal scaling of job consumers - Job Monitoring: Real-time dashboard for job status - Error Handling: Comprehensive error logging and alerting - Job Recovery: Automatic recovery from system failures

Push Sender Architecture¶

📱 Push Notification Delivery Service - Purpose: High-performance push notification delivery to mobile devices - Technology: Node.js service with FCM (Android) and APNs (iOS) integration - Architecture: Multi-tenant service with device token management - Features: Batch delivery, delivery tracking, token refresh, A/B testing

🔧 Core Components - Token Manager: Device token validation and refresh - Batch Processor: Efficient batch notification delivery - Delivery Tracker: Real-time delivery status tracking - Rate Limiter: Platform-specific rate limiting - Analytics Engine: Delivery performance analytics

📊 Platform Support

🔗 Integration Points - Notify Service: Notification request processing - Parse Server: Device token management - Appgain Server: User preferences and targeting - Analytics: Delivery performance tracking - Task Queue: Asynchronous delivery processing

📈 Performance Metrics - Delivery Rate: 95%+ successful delivery - Latency: < 2 seconds average delivery time - Throughput: 50,000+ notifications per minute - Uptime: 99.9% service availability

🔧 Configuration & Management - Platform Credentials: FCM and APNs certificate management - Device Token Sync: Automatic token refresh and cleanup - Delivery Optimization: Intelligent batching and retry logic - Monitoring: Real-time delivery status and error tracking - A/B Testing: Support for notification A/B testing

Shrinkit Shopify Backend Architecture¶

🛒 E-commerce Automation & Integration Platform - Purpose: Comprehensive Shopify application backend handling multiple e-commerce automation scenarios - Technology: Python Flask application with MongoDB and Redis - Architecture: Multi-tenant platform supporting multiple Shopify apps and integrations - Features: Abandoned cart recovery, dynamic pricing, lens recommendations, Odoo ERP integration

🔧 Core Application Types

🔗 Integration Points - Shopify GraphQL/REST APIs: Store data synchronization and management - Odoo XML-RPC: ERP integration for product/order synchronization - Appgain Platform: Automation workflows and customer engagement - Parse Server: Data persistence and multi-tenant storage - Redis: Caching and session management - AWS S3: File storage for product images and assets

📊 Key Business Scenarios

Abandoned Cart Recovery¶

• Process: Detects abandoned checkouts via Shopify webhooks → Stores data in Parse → Triggers automated campaigns via Appgain
• Value: Recovers 15-30% of lost sales, increases conversion rates

Dynamic Pricing Management¶

• Process: Monitors exchange rates → Updates product prices automatically → Maintains profit margins
• Value: Optimizes pricing for international markets, maximizes profits

Customer Lifecycle Management¶

• Process: Syncs customer data → Segments based on behavior → Triggers personalized campaigns
• Value: Improves customer retention, increases lifetime value

ERP Integration (Odoo)¶

• Process: Bidirectional sync between Shopify and Odoo → Product/order synchronization → Financial reconciliation
• Value: Eliminates data duplication, ensures consistency

🏗️ Technical Architecture

Core Components¶

• Flask Application: Main web framework with Blueprint-based routing
• Parse/MongoDB: Primary data store for store configurations and sync data
• Redis: Caching and session management
• Shopify GraphQL/REST APIs: External Shopify store interactions
• Odoo XML-RPC: ERP integration for product/order synchronization

Project Structure¶

shopify-backend/
├── api/                          # Main application code
│   ├── abandoned_checkouts/      # Abandoned cart automation
│   ├── core/                     # Core app installation & management
│   ├── gql/                      # GraphQL queries & mutations
│   ├── helpers/                  # Utility functions
│   ├── lens_recommender/         # Lens recommendation engine
│   ├── odoo/                     # Odoo ERP integration
│   ├── price_updater/            # Dynamic pricing system
│   ├── products/                 # Product management APIs
│   ├── smartlinks/               # URL shortening & tracking
│   ├── stores/                   # Store management
│   └── webhooks/                 # Shopify webhook handlers
├── instance/                     # Environment-specific config
├── tests/                        # Test suite
└── Dockerfile                    # Container configuration

🔐 Authentication Models 1. Appgain JWT (authtoken header) - Dashboard and admin requests 2. Shopify Bearer JWT (Authorization: Bearer <token>) - GraphQL proxy endpoint 3. Shopify Access Token (in-memory) - Internal Shopify API calls 4. No Authentication - Shopify webhooks, health checks

📊 Database Schema

Primary Collections¶

• shopifyActiveShops: Store configuration and credentials
• Users{store_subdomain}: Customer data per store
• Purchases{store_subdomain}: Order data per store
• Products{store_subdomain}: Product data per store
• Checkouts{store_subdomain}: Abandoned checkouts per store

Store Configuration Schema¶

{
  "_id": "ObjectId",
  "name": "store.myshopify.com",
  "type": "shrinkit|price_updater|lens_recommender|private_app",
  "accessToken": "shpat_...",
  "apiKey": "appgain_api_key",
  "appId": "appgain_app_id",
  "subdomain": "store_subdomain",
  "accountId": "user_account_id",
  "authtoken": "jwt_token",
  "userEmail": "store_owner@email.com",
  "otpEnabled": false,
  "v3Status": "enabled|disabled|in-progress|failed",
  "odoo": {
    "host": "odoo.example.com",
    "db": "database_name",
    "user": "username",
    "password": "password",
    "status": "synced|failed|in-progress",
    "last_sync_at": "2024-01-01T00:00:00Z"
  }
}

📈 Performance Metrics - Multi-Store Support: 100+ active Shopify stores - Webhook Processing: 1,000+ webhooks per hour - Data Sync: Real-time product/order synchronization - API Response Time: < 500ms average response time - Uptime: 99.9% service availability

🔧 Configuration & Management - Environment Variables: Shopify API credentials, database connections - Store Management: Dynamic store registration and configuration - Webhook Registration: Automatic webhook setup for new stores - Error Handling: Comprehensive error logging and alerting - Monitoring: Real-time performance and health monitoring

🚀 Deployment & Scaling - Containerized: Docker-based deployment - Multi-Environment: Development, staging, production - CI/CD: GitLab CI/CD pipeline with automated testing - Horizontal Scaling: Support for multiple instances - Load Balancing: Nginx-based load balancing

🔍 Security Considerations - Rate Limiting: API rate limiting for external requests - Input Validation: Comprehensive input validation and sanitization - Secret Management: Environment-based configuration management - Audit Logging: Complete audit trail for all operations - Data Privacy: GDPR compliance and data retention policies

Notify API Technical Documentation¶

🏗️ Code Structure & Architecture - Technology: Python Flask application with MongoDB - Repository: AppGain Notify Backend - Architecture: Modular Flask application with custom blueprints and utilities

Project Structure¶

appgain-notify/
├── instance/                          # Configuration and instance data
│   └── config.py                      # Database URL and configuration
├── app_notify/                        # Main application package
│   ├── __init__.py                    # Package initialization
│   ├── app.py                         # API routing and app creation
│   ├── config.py                      # Secret key configuration
│   ├── api/                           # API framework and utilities
│   │   ├── __init__.py                # API package initialization
│   │   ├── blueprint.py               # Custom Flask blueprint
│   │   ├── errors.py                  # Custom error handling
│   │   └── helpers.py                 # API helper functions
│   ├── app_parse/                     # Parse server integration
│   │   ├── __init__.py                # Parse package initialization
│   │   ├── parse.py                   # Parse data models
│   │   └── parse_bp.py                # Parse API endpoints
│   ├── apps_suit/                     # App suit management
│   │   ├── __init__.py                # Suit package initialization
│   │   └── suit.py                    # Suit data models
│   ├── utils/                         # Utility functions
│   │   ├── __init__.py                # Utils package initialization
│   │   ├── background.py              # Background task handling
│   │   ├── logging.py                 # Custom logging configuration
│   │   ├── validation.py              # Data validation utilities
│   │   ├── serialization.py           # JSON serialization helpers
│   │   └── random.py                  # Random number generation
│   ├── users/                         # User management
│   │   ├── __init__.py                # Users package initialization
│   │   ├── user.py                    # User data models
│   │   └── user_auth.py               # User authentication logic
│   ├── notify/                        # Notification core
│   │   ├── __init__.py                # Notify package initialization
│   │   ├── notify.py                  # Notification data models
│   │   └── notify_api.py              # Notification API endpoints
│   └── notification/                  # Notification management
│       ├── __init__.py                # Notification package initialization
│       ├── notifications.py           # Notification data models
│       └── notifications_api.py       # Notification management API
├── start                              # Application start script
├── stop                               # Application stop script
├── appgain_notify_wsgi.py             # WSGI entry point
├── run.py                             # Development server (port, debug)
├── requirements.txt                   # Python dependencies
├── .gitignore                         # Git ignore patterns
└── venv/                              # Virtual environment

Deployment Process¶

1. Repository Setup¶

# Clone the repository
git clone https://gitlab.com/appgain.io/appgain-notify.git /var/webapps/

# Navigate to project directory
cd /var/webapps/appgain-notify

2. Configuration Setup¶

# Create instance directory
mkdir instance

# Create configuration file
touch instance/config.py

# Edit configuration file
nano instance/config.py

3. Database Configuration¶

# instance/config.py
MONGODB_URI = ('mongodb://'
               '<username>:<password>@'
               '<host>'
               ':<port>'
               '/<database>')

4. Environment Setup¶

# Create virtual environment
virtualenv -p python3 venv

# Activate virtual environment
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Create logs directory
mkdir logs

# Deactivate virtual environment
deactivate

Execution & Runtime¶

Development Testing¶

# Activate virtual environment
source venv/bin/activate

# Set configuration path
export NOTIFICATION_CONFIG=config.py

# Run application
python run.py

# Deactivate virtual environment
deactivate

Production Deployment¶

# Start application in background
bash start

# Stop application
bash stop

CI/CD Configuration¶

Environment Variables¶

# Deployment Endpoints
API_ENDPOINT_PREPROD=https://preprod-api.appgain.io
API_ENDPOINT_PROD=https://api.appgain.io

# Authentication Tokens
API_TOKEN=ask your direct manager for the access
PREPROD_API_TOKEN=ask your direct manager for the access
PRERELEASE_API_TOKEN=ask your direct manager for the access
PROD_API_TOKEN=ask your direct manager for the access

# Authentication Types
PREPROD_AUTH_TYPE=basic
PRERELEASE_AUTH_TYPE=basic
PROD_AUTH_TYPE=basic

# Registry Configuration (GitLab CI/CD)
CI_REGISTRY_USER=$CI_REGISTRY_USER
CI_REGISTRY_PASSWORD=ask your direct manager for the access
CI_REGISTRY=$CI_REGISTRY
CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE

# Notifications
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

Deployment Locations¶

Deployment Workflow¶

1. Preprod: Push to dev branch triggers automatic deployment
2. Prerelease: Push to staging branch triggers automatic deployment
3. Production: Manual deployment from master branch

Key Features¶

API Endpoints¶

• Notification Management: Create, update, delete notifications
• Channel Integration: Multi-channel message delivery
• User Management: Authentication and user operations
• Parse Integration: Parse server data synchronization
• Suit Management: App suit configuration and management

Background Processing¶

• Asynchronous Tasks: Background task handling for heavy operations
• Message Queuing: Queue-based message processing
• Error Handling: Comprehensive error handling and logging
• Retry Logic: Automatic retry for failed operations

Monitoring & Logging¶

• Structured Logging: Comprehensive logging with correlation IDs
• Performance Monitoring: Response time and throughput tracking
• Error Tracking: Detailed error reporting and debugging
• Health Checks: Application health monitoring endpoints

Development Best Practices¶

Code Organization¶

• Modular Architecture: Clear separation of concerns
• Custom Blueprints: Reusable API components
• Utility Functions: Shared helper functions
• Data Models: MongoDB models with validation

Configuration Management¶

• Environment Variables: Secure configuration handling
• Instance Configuration: Instance-specific settings
• Secret Management: Secure secret key handling
• Database Configuration: Flexible database connection setup

Testing & Quality¶

• Unit Testing: Comprehensive test coverage
• Integration Testing: API endpoint testing
• Error Handling: Robust error handling patterns
• Documentation: Complete API documentation

Troubleshooting¶

Common Issues¶

1. Database Connection: Verify MongoDB URI and credentials
2. Port Conflicts: Check port configuration in run.py
3. Virtual Environment: Ensure proper venv activation
4. Dependencies: Verify all requirements are installed

Debug Commands¶

# Check application status
ps aux | grep python

# Monitor logs
tail -f logs/app.log

# Test database connection
python -c "from app_notify.app_parse.parse import db; print(db.list_collection_names())"

# Verify configuration
python -c "import os; print(os.environ.get('NOTIFICATION_CONFIG'))"

Performance Optimization¶

• Connection Pooling: Optimize database connections
• Caching: Implement Redis caching for frequently accessed data
• Background Tasks: Use background processing for heavy operations
• Monitoring: Set up performance monitoring and alerting

AppGain Server Backend Architecture¶

🏗️ Core Business Logic & Multi-Tenant Platform - Purpose: Centralized backend service for core business logic, CRM, analytics, and multi-tenant platform management - Technology: Node.js with Express.js framework and MongoDB - Architecture: Multi-tenant microservice architecture with RESTful APIs - Features: Business logic processing, user management, analytics, CRM functionality, multi-tenant isolation

🔧 Key Components - API Endpoints: RESTful APIs for core business operations - Business Logic: Core application logic and workflows - Multi-Tenant Management: Tenant isolation and resource management - CRM Integration: Customer relationship management functionality - Analytics Engine: Data analytics and reporting capabilities - Authentication: Multi-tenant user authentication and authorization - Rate Limiting: Tenant-specific rate limiting and throttling

📊 Service Capabilities

🔗 Integration Points - MongoDB: Primary data storage for all application data - Redis: Caching, session management, rate limiting - Parse Server: Mobile app backend services integration - Notify Service: Multi-channel communication - Automator Engine: Workflow automation triggers - External APIs: Third-party service integrations - Growth Machine: Analytics and ML processing

📈 Performance Metrics - Response Time: < 200ms average API response time - Throughput: 1000+ requests per second - Availability: 99.9% uptime - Error Rate: < 0.1% error rate - Multi-Tenant Isolation: Complete tenant data separation

🔧 Configuration & Management - Environment Variables: Comprehensive configuration management - Docker Support: Containerized deployment and scaling - CI/CD Pipeline: GitLab CI/CD with automated testing - Monitoring: Health checks and performance metrics - Logging: Structured logging with tenant correlation - Security: JWT authentication, tenant-based access control

CI/CD Configuration¶

Required Environment Variables¶

Deployment Variables¶

# API Endpoints
API_ENDPOINT_PREPROD=https://preprod-api.appgain.io
API_ENDPOINT_PROD=https://api.appgain.io

# Authentication Tokens
API_TOKEN=ask your direct manager for the access
PREPROD_API_TOKEN=ask your direct manager for the access
PRERELEASE_API_TOKEN=ask your direct manager for the access
PROD_API_TOKEN=ask your direct manager for the access

Authentication Customization¶

# Authentication Types (default: basic)
PREPROD_AUTH_TYPE=basic
PRERELEASE_AUTH_TYPE=basic
PROD_AUTH_TYPE=basic

# Alternative: Bearer token authentication
PREPROD_AUTH_TYPE=bearer
PRERELEASE_AUTH_TYPE=bearer
PROD_AUTH_TYPE=bearer

Registry Variables (GitLab CI/CD)¶

# Docker Registry Configuration
CI_REGISTRY_USER=$CI_REGISTRY_USER
CI_REGISTRY_PASSWORD=ask your direct manager for the access
CI_REGISTRY=$CI_REGISTRY
CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE

Notification Variables¶

# Deployment Notifications
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

Deployment Locations¶

Deployment Workflow¶

1. Preprod Deployment
2. Trigger: Push to dev branch
3. Location: ovh-preprod
4. Purpose: Testing and validation

5. Auto-deploy: Yes

6. Prerelease Deployment

7. Trigger: Push to staging branch
8. Location: ovh-devops
9. Purpose: Staging and pre-production testing

10. Auto-deploy: Yes

11. Production Deployment

12. Trigger: Manual deployment from master branch
13. Locations:
    • Appgain: ovh-parse-server, ovh-pushsender
    • Shrinkit: ovh-shrinkit
    • Frontend: ovh-devops
14. Purpose: Live production services
15. Auto-deploy: No (manual approval required)

Development Best Practices¶

Multi-Tenant Architecture¶

• Tenant Isolation: Complete data separation between tenants
• Resource Management: Tenant-specific resource allocation
• Access Control: Tenant-based authentication and authorization
• Data Partitioning: Database-level tenant isolation

Code Organization¶

• Modular Architecture: Clear separation of concerns
• API Design: RESTful APIs with consistent patterns
• Error Handling: Comprehensive error handling and logging
• Validation: Input validation and sanitization

Security Implementation¶

• JWT Authentication: Secure token-based authentication
• Tenant Security: Tenant-specific access controls
• Rate Limiting: Tenant-based rate limiting
• Data Encryption: Sensitive data encryption at rest and in transit

Troubleshooting¶

Common Issues¶

1. Environment Variables: Verify all required variables are set
2. API Endpoints: Check endpoint availability and connectivity
3. Authentication: Ensure proper authentication for deployment
4. Tenant Isolation: Verify tenant data separation
5. Database Connections: Check MongoDB connectivity and permissions

Debug Commands¶

# Check application status
docker ps | grep appgain-server

# Monitor logs
docker logs -f appgain-server-container

# Test API endpoints
curl -X GET "https://api.appgain.io/health" \
  -H "Authorization: Bearer $API_TOKEN"

# Verify environment variables
docker exec appgain-server-container env | grep API_

# Check tenant isolation
curl -X GET "https://api.appgain.io/tenants" \
  -H "Authorization: Bearer $API_TOKEN"

Performance Optimization¶

• Database Indexing: Optimize MongoDB indexes for tenant queries
• Caching Strategy: Implement Redis caching for frequently accessed data
• Connection Pooling: Optimize database connection pools
• Load Balancing: Implement proper load balancing for high availability

Recommended Improvements¶

Security Enhancements¶

• Environment-Specific Tokens: Use different tokens for each environment
• Granular Access Controls: Implement more detailed permission systems
• Token Rotation: Regularly rotate deployment and API tokens
• Audit Logging: Comprehensive audit trail for all operations

Monitoring & Observability¶

• Health Checks: Implement comprehensive health check endpoints
• Metrics Collection: Collect and monitor key performance metrics
• Alerting: Set up automated alerting for critical issues
• Logging: Structured logging with tenant correlation

Scalability¶

• Horizontal Scaling: Design for horizontal scaling across multiple instances
• Database Sharding: Implement database sharding for large-scale deployments
• Microservices: Consider breaking down into smaller microservices
• Caching Layers: Implement multi-level caching strategies

AppBackend Server Architecture¶

🏗️ Application Backend Services - Purpose: Centralized backend services for application logic and business operations - Technology: Node.js with Express.js framework - Architecture: Microservice-based architecture with RESTful APIs - Features: Business logic processing, data validation, service orchestration

🔧 Key Components - API Endpoints: RESTful APIs for application operations - Business Logic: Core application logic and workflows - Data Processing: Data transformation and validation - Service Integration: Integration with other microservices - Authentication: User authentication and authorization - Rate Limiting: API rate limiting and throttling

📊 Service Capabilities

🔗 Integration Points - MongoDB: Primary data storage for application data - Central Redis: Caching, session management, message queuing - Parse Server: Mobile app backend services - Notify Service: Multi-channel communication - Automator Engine: Workflow automation triggers - External APIs: Third-party service integrations

📈 Performance Metrics - Response Time: < 200ms average API response time - Throughput: 1000+ requests per second - Availability: 99.9% uptime - Error Rate: < 0.1% error rate

🔧 Configuration & Management - Environment Variables: Configuration management - Logging: Structured logging with correlation IDs - Monitoring: Health checks and performance metrics - Security: JWT authentication, rate limiting, CORS - Documentation: AppBackend Server SRS

Central Redis Architecture¶

🏗️ Centralized Caching & Message Queuing - Purpose: Centralized Redis instance for caching, session management, and message queuing - Technology: Redis with clustering and persistence - Architecture: High-availability Redis cluster with master-slave replication - Features: Caching, session storage, message queuing, pub/sub

🔧 Key Components - Cache Layer: Application data caching for improved performance - Session Store: User session management across services - Message Queue: Asynchronous message processing - Pub/Sub: Real-time event broadcasting - Rate Limiting: API rate limiting storage - Job Queue: Background job processing

📊 Redis Capabilities

🔗 Service Integration - Appgain Server: Session management, caching - AppBackend Server: Data caching, message queuing - Parse Server: Session storage, caching - Notify Service: Message queuing, rate limiting - Automator Engine: Job queuing, event broadcasting - Admin Server: Session management, caching

📈 Performance Metrics - Latency: < 1ms for cache operations - Throughput: 100,000+ operations per second - Memory Usage: Optimized for high-performance caching - Persistence: RDB and AOF persistence for data durability - Replication: Master-slave replication for high availability

🔧 Configuration & Management - Clustering: Redis cluster for horizontal scaling - Persistence: RDB snapshots and AOF logging - Security: Authentication and network security - Monitoring: Redis INFO commands and metrics - Backup: Automated backup and recovery procedures

📚 Best Practices - Key Naming: Consistent key naming conventions - TTL Management: Appropriate expiration times for cached data - Memory Management: Memory usage monitoring and optimization - Connection Pooling: Efficient connection management - Error Handling: Graceful handling of Redis failures

Admin Server Architecture¶

🏗️ Comprehensive Management & Analytics Platform - Purpose: Centralized backend solution for managing administrative functions, mobile app lifecycle, user authentication, analytics tracking, and system monitoring - Technology: Flask-based Python application with MongoDB and Redis - Architecture: Modular microservice architecture with RESTful APIs - Features: Complete app management, analytics, notifications, deployment automation

🔧 Key Components - User Management: Complete authentication, authorization, and profile management - App Suit Management: Mobile app lifecycle management with deep linking and analytics - Smart Links & Landing Pages: Dynamic link generation and landing page management - Notification Services: Multi-channel notification delivery (Email, SMS, WhatsApp) - Analytics & Reporting: Comprehensive analytics and conversion tracking - System Monitoring: Real-time system health monitoring and alerting - Deployment Management: Automated deployment and infrastructure management

📊 Service Capabilities

🔗 Integration Points - MongoDB: Primary data storage for all application data - Redis: Session management, caching, message queuing - Parse Server: Mobile app backend services integration - Notify Service: Multi-channel communication - Vercel: Frontend deployment and hosting - AWS: File upload and CDN management - Stripe: Subscription and billing management - Sentry: Error tracking and debugging - Prometheus: Application performance monitoring

📈 Performance Metrics - Response Time: < 200ms average API response time - Throughput: 1000+ requests per second - Availability: 99.9% uptime - Error Rate: < 0.1% error rate - Database Performance: < 50ms query response time

🔧 Configuration & Management - Environment Variables: Comprehensive configuration management - Docker Support: Containerized deployment and scaling - CI/CD Pipeline: GitLab CI/CD with automated testing - Monitoring: Prometheus metrics, Sentry error tracking - Logging: Structured logging with correlation IDs - Security: JWT authentication, role-based access control

🏗️ Project Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Client Apps   │    │   Mobile Apps   │    │   Web Dashboard │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌─────────────▼─────────────┐
                    │   AppGain Admin Server    │
                    │   (Flask Application)     │
                    └─────────────┬─────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          │                       │                       │
┌─────────▼─────────┐  ┌─────────▼─────────┐  ┌─────────▼─────────┐
│   MongoDB Atlas   │  │   Redis Cache     │  │   External APIs   │
│   (Primary DB)    │  │   (Session/Cache) │  │   (Stripe, AWS)   │
└───────────────────┘  └───────────────────┘  └───────────────────┘

📁 Core Modules - admin_server/: Main application package - api/: API framework and utilities - apps_suit/: App suit management - users/: User management and authentication - app_plans/: Subscription plans - app_smartlinks/: Smart links management - app_landingpages/: Landing pages management - notify/: Notification services - vercel_deployment/: Vercel deployment automation - system_monitor/: System monitoring and health checks - utils/: Utility functions and helpers

🚀 Advanced Features - Vercel Integration: Automated frontend deployment and management - AWS Integration: File upload and CDN management - Prometheus Monitoring: Application performance monitoring - Sentry Integration: Error tracking and debugging - CI/CD Pipeline: Automated testing and deployment - Docker Support: Containerized deployment and scaling - WhatsApp Integration: WhatsApp Business API integration - Background Tasks: Asynchronous processing with threading

🔧 Development & Deployment - Code Style: Black formatting, PEP 8 guidelines - Testing: pytest with comprehensive test coverage - Database: MongoFrames ODM for MongoDB - Background Tasks: Threading-based background processing - API Design: RESTful APIs with custom Flask blueprints - Error Handling: Comprehensive error handling and logging

📚 Documentation & Resources - API Documentation: Complete REST API reference - Deployment Guide: Docker and CI/CD setup - Development Guide: Local setup and development workflow - Troubleshooting: Common issues and solutions - GitLab Repository: AppGain Admin Server

Email Platform Architecture¶

🏗️ Mailu-Based Email Infrastructure - Purpose: Self-hosted email server replacing AWS SES - Technology: Mailu Docker stack with multi-tenant support - Architecture: Dedicated stack per tenant or shared multi-domain setup - Features: SMTP, IMAP, POP3, REST API, SSL, DKIM, DMARC

🔧 Key Components - Mailu Server: All-in-one email solution (Postfix + Dovecot) - DNS Automation: DigitalOcean API integration for domain management - Email Warm-up: Automated reputation building system - Bounce Tracking: Webhook integration with CRM - Monitoring: Promtail → Loki → Grafana pipeline

Monitoring & Observability¶

Centralized Logging System¶

Loki is our centralized log aggregation system that collects, stores, and queries logs from all backend services. It's designed for high availability and scalability, making it perfect for our microservices architecture.

Key Features of Loki:¶

• High Performance: Efficient log storage and querying
• Scalability: Handles large volumes of log data
• Cost-Effective: Optimized storage for log data
• Real-time Queries: Fast log search and filtering
• Integration: Works seamlessly with Grafana for visualization

Log Sources:¶

• Appgain Server: Main business logic and CRM functionality
• Parse Server: Backend-as-a-Service for mobile apps
• Notify Service: Multi-channel notification system
• Automator Engine: Workflow automation and triggers
• Admin Server: Management interface and analytics
• API Gateway: Request routing and authentication
• Task Queue: Background job processing

Accessing Microservice Logs:¶

Grafana Explore - Microservice Logs

Monitoring Tools¶

Essential Documentation Links¶

🎥 Video Resources & Tutorials¶

System Architecture Videos¶

Automator Journey Builder¶

Platform Architecture: Suit Types¶

Overview¶

Appgain operates with two distinct types of suits that have different architectural patterns and deployment models. Understanding these is crucial for effective troubleshooting, testing strategies, customer support, and system monitoring.

Two Types of Suits¶

1. App Boost Suits (Mobile App Focus)¶

Platform: Appgain only User Management: Self-registered users (created by support/sales team ONLY) Examples: Spinneys, Ikhair, Ihsan

Architecture Characteristics: - Dedicated Parse Server: Each suit has its own Parse Server instance - Dedicated Database: Each suit has its own Parse DB instance - Isolated Cloud Functions: Each suit has its own CFs and unique starting point file (main.js) - Mobile SDK Integration: Each suit represents a connected mobile app using Appgain Mobile SDK - Standard Collections: Each suit has dedicated DB instance with similar collections (i.e., _User, AppSegments, etc.) - Full Communication: Each suit can send app push, web push, SMS, email, WhatsApp - Naming Convention: Suits are called "Projects"

Support & Testing Implications: - Isolated Issues: Problems in one suit don't affect others - Dedicated Monitoring: Individual monitoring and alerting per suit - Custom Configurations: Suit-specific settings and configurations - Independent Troubleshooting: Can troubleshoot each suit separately - Suit-specific Documentation: Separate troubleshooting guides per suit - Isolated Testing: Each suit can be tested independently - Dedicated Test Environments: Separate test environments per suit - Custom Test Cases: Suit-specific test scenarios and requirements - Independent Test Execution: Can run tests for each suit separately - Suit-specific Test Data: Individual test data sets per suit

2. App Marketing Suits (Web/Store Focus)¶

Platforms: Appgain (retail-gain/charity projects) and Shrinkit (store integrations) User Management: - Appgain: Self-registered users (created by support/sales team ONLY) - Shrinkit: External integration users (created by partner integrations ONLY - Shopify, Salla, Zid)

Architecture Characteristics: - Shared Parse Server: All suits share the same Parse Server instance - Appgain: Called "core" Parse Server - Shrinkit: Called "shrnkt" Parse Server - Shared Cloud Functions: All suits share the same CFs and same starting point file (main.js) - Web/Store Integration: All suits are linked to websites, stores, charity pages, etc., syncing data via webhooks or API calls - Shared Database with Namespacing: All suits share similar database, but each suit has namespaced collections: {TABLE}{SUBDOMAIN} - Example: Userstestawy (table=Users for subdomain=testawy) - Limited Communication: Can send web push, SMS, email, WhatsApp; NO app push (no mobile SDK)

Support & Testing Implications: - Shared Issues: Problems can affect multiple suits simultaneously - Coordinated Monitoring: Shared monitoring with suit-specific breakdowns - Standardized Configurations: Common settings across all suits - Cross-suit Troubleshooting: Issues may require investigation across multiple suits - Shared Documentation: Common troubleshooting guides with suit-specific variations - Shared Testing: Tests affect all suits simultaneously - Coordinated Test Environments: Shared test environment with suit isolation - Standardized Test Cases: Common test scenarios across all suits - Cross-suit Test Execution: Tests may need to run across multiple suits - Shared Test Data: Common test data with suit-specific variations

Key Differences Summary¶

Foundation Courses (Required for All Roles)¶

• 🔧 Version Control: Git, GitHub and GitHub Actions
• 🗃️ Database Fundamentals: MongoDB Essentials
• 🔗 API Basics: Learn API Development with Flask MySQL in Python

Server Management Best Practices¶

• Connection: Use autossh for stable connections with keep-alive
• Monitoring: All servers have ServerAliveInterval 60 and ServerAliveCountMax 3
• Access: Ubuntu user access with SSH key authentication
• Backup: Regular database backups and file system snapshots
• Security: Firewall rules, SSL certificates, and access controls

🏗️ This common knowledge base provides the foundation that all team members need to understand our system architecture and work effectively together.

← Back to Home