Skip to content

🔧 Backend Engineer - Complete Guide

Specialized guide for Backend Engineers at Appgain

🎯 Role Overview

Backend Engineers at Appgain focus on developing and maintaining server-side applications, APIs, and database systems that power our multi-tenant SaaS platform.

📚 Specialized Learning

Required Courses

🐍 Python Flask Training

  • Flask Framework: Learn Python Flask for backend development
  • RESTful APIs: Build REST APIs with Flask
  • Database Integration: Connect Flask with MongoDB
  • Authentication: Implement user authentication and authorization
  • Testing: Write unit tests for Flask applications
  • AppGain Microservice Development Blueprint: Complete guide to building microservices following AppGain standards

💻 Cursor.com Training

  • AI-Powered Coding: Master Cursor.com for enhanced development productivity
  • Code Generation: Learn to use AI assistance for faster coding
  • Refactoring: Use AI to improve and refactor existing code
  • Debugging: Leverage AI for better debugging and problem-solving
  • Best Practices: Understand when and how to use AI assistance effectively

📡 Postman Training

  • API Testing: Master API testing and automation with Postman
  • Collection Management: Organize and manage API collections
  • Environment Variables: Use environments for different deployment stages
  • Automation: Create automated test suites and CI/CD integration
  • Documentation: Generate and maintain API documentation

🗄️ MongoDB Training

  • Database Design: Learn MongoDB schema design best practices
  • Query Optimization: Master MongoDB query performance and indexing
  • Aggregation Pipeline: Use MongoDB aggregation for complex data operations
  • Data Modeling: Design efficient data models for different use cases
  • Performance Tuning: Optimize MongoDB performance and scalability

Additional Resources

🛠️ Backend Infrastructure

Core Services

  • Appgain Server: Main business logic and CRM functionality
  • Parse Server: Backend-as-a-Service for mobile apps
  • Notify Service: Multi-channel notification system
  • Automator Engine: Workflow automation and triggers
  • Admin Server: Management interface and analytics

Database Architecture

  • MongoDB: Primary data storage for applications
  • PostgreSQL: KONG configuration and admin data
  • Redis: Caching and message queuing
  • MongoDB Clusters: Replicated database setup

🔧 Key Responsibilities

1. API Development

  • RESTful APIs: Design and implement REST APIs
  • GraphQL: Develop GraphQL schemas and resolvers
  • API Documentation: Maintain comprehensive API docs
  • Versioning: Manage API versioning and backward compatibility

2. Database Management

  • Schema Design: Design efficient database schemas
  • Query Optimization: Optimize database queries for performance
  • Data Migration: Plan and execute database migrations
  • Backup & Recovery: Ensure data integrity and recovery

3. System Integration

  • Third-party APIs: Integrate external services and APIs
  • Microservices: Develop and maintain microservice architecture
  • Message Queues: Implement asynchronous processing
  • Webhooks: Handle real-time data synchronization

4. Performance & Scalability

  • Load Balancing: Implement horizontal scaling
  • Caching: Design and implement caching strategies
  • Monitoring: Set up performance monitoring and alerting
  • Optimization: Continuously optimize system performance

🚀 Technical Stack

Core Technologies

  • Node.js: Primary runtime environment
  • Express.js: Web application framework
  • MongoDB: NoSQL database
  • Redis: In-memory data store
  • Parse Server: Backend-as-a-Service framework

Development Tools

  • Git: Version control
  • Postman: API testing and documentation
  • Docker: Containerization
  • ESLint: Code quality and linting

📊 Success Metrics

Performance Metrics

  • API Response Time: < 200ms average
  • Database Query Time: < 50ms average
  • System Uptime: 99.9% availability
  • Error Rate: < 0.1% error rate

Code Quality Metrics

  • Test Coverage: > 80% code coverage
  • Code Review: 100% code review compliance
  • Documentation: Complete API documentation
  • Security: Zero critical security vulnerabilities

🔗 Integration Points

System Integration

  • Frontend Applications: React, Next.js applications
  • Mobile Apps: iOS and Android applications
  • External Services: Payment gateways, email providers
  • Monitoring: Prometheus, Grafana, Loki

📊 Backend Servers Logging

Loki Log Aggregation System

Loki is our centralized log aggregation system that collects, stores, and queries logs from all backend services. It's designed for high availability and scalability, making it perfect for our microservices architecture.

Key Features of Loki:

  • High Performance: Efficient log storage and querying
  • Scalability: Handles large volumes of log data
  • Cost-Effective: Optimized storage for log data
  • Real-time Queries: Fast log search and filtering
  • Integration: Works seamlessly with Grafana for visualization

Log Sources:

  • Appgain Server: Main business logic and CRM functionality
  • Parse Server: Backend-as-a-Service for mobile apps
  • Notify Service: Multi-channel notification system
  • Automator Engine: Workflow automation and triggers
  • Admin Server: Management interface and analytics
  • API Gateway: Request routing and authentication
  • Task Queue: Background job processing

Accessing Microservice Logs

To get logs for any microservice, use the following Grafana Explore link:

Grafana Explore - Microservice Logs

How to Use:

  1. Replace Service Name: Change admin-server in the URL to any service name:
  2. appgain-server
  3. parse-server
  4. notify-service
  5. automator-engine
  6. api-gateway

  7. task-queue

  8. Time Range: Adjust the time range in the URL:

  9. now-1h for last hour
  10. now-6h for last 6 hours

  11. now-24h for last 24 hours

  12. Query Examples: 

    # All logs for appgain-server
{service="appgain-server"}

# Error logs for notify-service
{service="notify-service"} |= "error"

# Specific API endpoint logs
{service="api-gateway"} |= "/api/v1/users"

Logging Best Practices

Structured Logging

// Good logging practice
logger.info('User authentication successful', {
  userId: user.id,
  service: 'auth-service',
  timestamp: new Date().toISOString(),
  requestId: req.headers['x-request-id']
});

// Error logging with context
logger.error('Database connection failed', {
  error: error.message,
  service: 'appgain-server',
  database: 'mongodb',
  timestamp: new Date().toISOString()
});

Log Levels

  • DEBUG: Detailed information for debugging
  • INFO: General information about application flow
  • WARN: Warning messages for potential issues
  • ERROR: Error messages for failed operations
  • FATAL: Critical errors that may cause system failure

Video Resources

Backend Logging Training Video

Data Sources

  • User Data: Authentication and user management
  • Business Logic: CRM, analytics, reporting
  • External APIs: Third-party integrations
  • File Storage: AWS S3, local file systems

📋 Daily Operations

Morning Routine

# Check system health
curl http://api.appgain.io/health

# Monitor database performance
mongo --eval "db.stats()"

# Check service logs
docker logs appgain-server

# Review error rates
curl http://monitor.instabackend.io:9090/a../query?query=error_rate

Development Workflow

# Start development environment
docker compose up -d

# Run tests
npm test

# Start development server
npm run dev

# Deploy to staging
git push origin staging

🚀 Deployment Process

Automatic Deployment: - Usually, deployment to production happens automatically when you push to the master branch - CI/CD pipeline handles the build and deployment process

Manual Deployment: - You can perform manual deployment and Docker image builds from https://status.instabackend.io/ - This is useful for hotfixes or when you need to bypass the automatic pipeline

Monitoring & Maintenance

# Monitor API performance
curl http://monitor.instabackend.io:9090/a../query?query=api_response_time

# Check database connections
mongo --eval "db.serverStatus().connections"

# Backup database
mongodump --out /backup/$(date +%Y%m%d)

🎯 Project Examples

1. Multi-tenant API Development

  • Goal: Scalable APIs for multiple clients
  • Technology: Node.js, Express, MongoDB
  • Integration: KONG API Gateway
  • Metrics: Response time, throughput, error rate

2. Real-time Notification System

  • Goal: Instant message delivery across channels
  • Technology: WebSockets, Redis, FCM/APNs
  • Integration: Notify Service
  • Metrics: Delivery rate, latency, user engagement

3. Data Analytics Pipeline

  • Goal: Real-time data processing and analytics
  • Technology: Node.js streams, MongoDB aggregation
  • Integration: Growth Machine
  • Metrics: Processing speed, data accuracy, insights quality

📊 Monitoring & Error Tracking

🔍 System Status & Monitoring

🌐 Appgain Status Page

  • URL: https://status.instabackend.io/
  • Purpose: Real-time system status and service health
  • Access: Public - No authentication required
  • Features: Service uptime, incident history, performance metrics

📊 Centralized Logging (Loki)

🔍 Sentry Accounts

Sentry is our primary error tracking and performance monitoring platform. Backend engineers need access to monitor and debug issues across all our services.

🔗 Sentry Access

1. 🛒 Shrinkit Sentry Account

Email: mohamed.shaheen@appgain.io

Components: - 🖥️ Shrinkit Server - Main e-commerce backend services - 📢 Shrinkit Notify - Multi-channel notification system - 🤖 Shrinkit Automator - Workflow automation engine - 🏢 Shrinkit Admin - Administrative interface - 🛍️ Shopify Backend - E-commerce integration services - ⚙️ Engine Task-Queue - Background job processing - 📱 Engine Push-Sender - Push notification delivery

Key Metrics to Monitor: - Error rates and types - API response times - Database query performance - Background job success rates - Push notification delivery rates

2. 🎨 Appgain Frontend Sentry Account

Email: apps@appgain.io

Components: - 📊 Appgain Dashboard - Main platform dashboard - 🛒 Shrinkit Dashboard - E-commerce management interface - 📱 Mobile SDK - Mobile application SDK

Key Metrics to Monitor: - Frontend error rates - User interface performance - Mobile SDK integration issues - Dashboard loading times - User interaction patterns

3. 🩺 iKhair Sentry Account

Email: reach@ikhair.net

Components: - 🩺 iKhair API/Reports - Donation platform APIs - 🎗️ Jonation API - Donation processing services - 🏥 Ihsan API - Healthcare integration services

Key Metrics to Monitor: - API response times - Donation processing success rates - Donation data integration - Report generation performance - Payment processing errors

4. ⚙️ Appgain Backend Sentry Account

Email: apps+be@appgain.io

Components: - 🖥️ Appgain Server - Core business logic and CRM - 📢 Notify - Multi-channel messaging system - 🤖 Automator - Workflow automation engine - 🏢 Admin Server - Management interface - 🔧 AppBackend Server - Application backend services

Key Metrics to Monitor: - Server performance and uptime - API endpoint response times - Database query optimization - Background job processing - System integration health

🔍 Sentry Monitoring Best Practices

Daily Monitoring Routine

# Check error rates across all services
# Access: https://sentry.io/auth/login/

# 1. Review critical errors (P0, P1)
# 2. Check performance metrics
# 3. Monitor API response times
# 4. Review background job success rates
# 5. Check database query performance

Error Response Process

  1. 🚨 Critical Errors (P0): Immediate response required
  2. System down or major functionality broken
  3. Contact DevOps team immediately

  4. Create incident ticket

  5. ⚠️ High Priority (P1): Response within 1 hour

  6. Significant functionality impacted
  7. Multiple users affected

  8. Create bug ticket and assign

  9. 📊 Medium Priority (P2): Response within 4 hours

  10. Minor functionality issues
  11. Performance degradation

  12. Create bug ticket

  13. 📝 Low Priority (P3): Response within 24 hours

  14. Cosmetic issues
  15. Minor performance issues
  16. Create enhancement ticket

Performance Monitoring

// Example: Monitor API response times
const performanceMetrics = {
  'api_response_time': {
    'target': '< 200ms',
    'alert_threshold': '> 500ms',
    'critical_threshold': '> 1000ms'
  },
  'database_query_time': {
    'target': '< 50ms',
    'alert_threshold': '> 100ms',
    'critical_threshold': '> 200ms'
  },
  'error_rate': {
    'target': '< 0.1%',
    'alert_threshold': '> 1%',
    'critical_threshold': '> 5%'
  }
};

Debugging with Sentry

// Example: Proper error tracking
const Sentry = require('@sentry/node');

// Initialize Sentry
Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 1.0,
});

// Track custom errors
try {
  // Your code here
} catch (error) {
  Sentry.captureException(error, {
    tags: {
      service: 'appgain-server',
      endpoint: '/api/users',
      user_id: userId
    },
    extra: {
      request_data: req.body,
      user_context: userContext
    }
  });
  throw error;
}

// Track performance
Sentry.startTransaction({
  op: 'http.server',
  name: 'GET /api/users',
}).finish();

🚨 Alert Configuration

Critical Alerts

  • System Down: Immediate notification to DevOps team
  • High Error Rate: Alert if error rate > 5%
  • Performance Degradation: Alert if response time > 1000ms
  • Database Issues: Alert on connection failures

Warning Alerts

  • Increased Error Rate: Warning if error rate > 1%
  • Performance Issues: Warning if response time > 500ms
  • Memory Usage: Warning if memory usage > 80%
  • Disk Space: Warning if disk usage > 85%

Information Alerts

  • New Error Types: Track new error patterns
  • Performance Trends: Monitor performance over time
  • User Impact: Track user-affected errors
  • Integration Issues: Monitor external service errors

📊 Reporting & Analytics

Daily Reports

  • Error rate summary by service
  • Performance metrics overview
  • Critical issues resolved
  • New error patterns identified

Weekly Reports

  • Error trend analysis
  • Performance improvement opportunities
  • Service health scores
  • User impact assessment

Monthly Reports

  • Overall system health
  • Performance optimization recommendations
  • Error pattern analysis
  • Proactive monitoring improvements

🔧 Troubleshooting with Sentry

Common Debugging Scenarios

1. High Error Rate Investigation

# Steps to investigate high error rates
1. Check Sentry for error patterns
2. Identify affected services
3. Review recent deployments
4. Check database performance
5. Monitor external dependencies

2. Performance Issue Analysis

# Steps to analyze performance issues
1. Review response time metrics
2. Check database query performance
3. Monitor memory usage
4. Analyze CPU utilization
5. Review caching effectiveness

3. Integration Error Debugging

# Steps to debug integration errors
1. Check API endpoint health
2. Verify authentication tokens
3. Review request/response logs
4. Test external service connectivity
5. Validate data format compatibility

Debug Commands

# Check Sentry integration
curl -H "X-Sentry-Auth: Sentry sentry_version=7" \
  https://sentry.io/api/0/projects/

# Monitor error rates
curl -X GET "https://sentry.io/api/0/projects/{org}/{project}/stats/" \
  -H "Authorization: Bearer {token}"

# Check performance metrics
curl -X GET "https://sentry.io/api/0/projects/{org}/{project}/events/" \
  -H "Authorization: Bearer {token}"

📚 Learning Resources

Sentry Documentation

Best Practices

🎯 Quick Navigation

⚙️ Backend Engineers build the robust server-side infrastructure that powers our platform's core functionality and business logic.

← Back to Home | ← Previous: Common Knowledge

Ask Chehab GPT