# RSC Revolution Market Analytics System

Automated market data collection and analysis system for RSC Revolution private server.

## Overview

This system provides real-time market analysis, price tracking, and trading insights by collecting data from the RSC Revolution game database and generating comprehensive analytics.

**Key Features:**
- Real-time market state tracking
- Historical price analysis
- Notable trade detection
- Automated data collection
- Performance monitoring
- Cross-platform deployment support

## Quick Start

### Installation
```bash
# 1. Clone repository
git clone <your-repo-url> market_analytics
cd market_analytics

# 2. Install dependencies
pip3 install -r requirements.txt

# 3. Configure environment
cp .env.example .env
nano .env  # Edit database credentials

# 4. Test installation
python3 tests/test_quick.py

# 5. Setup automated collection
chmod +x setup_cron_template.sh
./setup_cron_template.sh
```

### Testing
```bash
# Quick smoke test (< 30 seconds)
python3 tests/test_quick.py

# Comprehensive functionality test
python3 tests/test_essential.py

# Production readiness validation
python3 tests/test_deployment.py
```

## Directory Structure
```
market_analytics/
├── collectors/          # Data collection scripts
│   ├── current_state.py    # Every 5 min - market state
│   ├── snapshot.py  # Hourly price snapshots
│   └── notable_trades.py   # High-value trade detection
├── aggregators/        # Data aggregation scripts
│   ├── hourly_aggregator.py  # Hourly rollup from snapshots
│   ├── daily_aggregator.py   # Daily OHLC statistics
│   └── monthly_aggregator.py # Monthly summaries
├── analyzers/          # Analysis and trending scripts
│   ├── calculate_trending.py  # Price trend analysis
│   ├── volatility_analyzer.py # Market volatility
│   └── run_analysis.py       # Analysis runner
├── maintenance/        # Cleanup and maintenance
│   ├── cleanup_old_data.py   # Data retention
│   ├── backup_tables.py      # Database backups
│   └── log_rotation.py       # Log management
├── monitoring/         # Health checks and alerts
│   ├── health_check.py       # System health
│   └── performance_check.py  # Performance metrics
├── config/            # Configuration and utilities
│   ├── database.py          # Database connections
│   ├── paths.py            # Dynamic path detection
│   └── logging_config.py    # Logging setup
├── tests/             # Comprehensive test suite
│   ├── test_quick.py        # Fast smoke tests
│   ├── test_essential.py    # Core functionality
│   ├── test_deployment.py   # Production readiness
│   ├── test_integration.py  # Database integration
│   ├── test_collectors.py   # Unit tests
│   └── test_analyzers.py    # Analyzer tests
├── tools/             # Development and utility tools
│   ├── backfill_hourly.py    # One-time hourly data backfill
│   ├── import_historical.py  # Historical data import
│   └── run_manual_tests.py   # Manual test runner
├── docs/              # Documentation
└── logs/              # Application logs (auto-created)
```

## Automated Collection Schedule

The system runs continuous data collection via cron jobs:

- **Every 15 minutes**: Market price snapshots
- **Every 5 minutes** (offset by 1 min): Update current market state
- **Every hour at :02**: Hourly data aggregation
- **Every hour at :05**: Collect notable trades
- **Every 10 minutes**: Health check monitoring
- **Daily (00:05)**: Aggregate daily statistics
- **Daily (01:00)**: Clean up old data
- **Daily (01:30)**: Log rotation
- **Every 2 hours**: Market trend analysis

## Database Tables

### Analytics Tables (Created by system)
- `market_current_state` - Real-time market state and prices
- `market_snapshots` - 15-minute price snapshots (14-day retention)
- `market_prices_hourly` - Hourly aggregated data (180-day retention)
- `market_prices_daily` - Daily OHLC and summary statistics (permanent)
- `market_notable_trades` - High-value trades and market events

### Game Tables (Read-only access required)
- `pk_market_sales` - Sale transaction history
- `pk_market_active` - Active market listings
- `pk_market_inactive` - Completed/expired listings
- `pk_itemdef` - Item definitions and metadata
- `pk_players` - Player information

## Configuration

### Environment Variables (.env file)
```env
# Database Configuration
DB_HOST=localhost
DB_USER=analytics_user
DB_PASSWORD=your_password
DB_NAME=rscs_main

# Optional Settings
LOG_LEVEL=INFO
BACKUP_RETENTION_DAYS=30
```

### Database Permissions
The analytics user needs SELECT access on game tables:
```sql
GRANT SELECT ON rscs_main.pk_market_* TO 'analytics_user'@'%';
GRANT SELECT ON rscs_data.pk_itemdef TO 'analytics_user'@'%';
GRANT SELECT ON rscs_main.pk_players TO 'analytics_user'@'%';
GRANT ALL ON rscs_main.market_* TO 'analytics_user'@'%';
```

## Manual Operations

### Check System Status
```bash
# Quick health check
python3 monitoring/health_check.py

# Performance metrics
python3 monitoring/performance_check.py

# View recent data
python3 -c "
from config.database import get_db_connection
with get_db_connection() as conn:
    cursor = conn.cursor()
    cursor.execute('SELECT COUNT(*) FROM market_current_state')
    print(f'Items tracked: {cursor.fetchone()[0]}')
"
```

### Manual Data Collection
```bash
# Update current market state
python3 collectors/current_state.py

# Create price snapshot
python3 collectors/snapshot.py

# Find notable trades
python3 collectors/notable_trades.py
```

### Run Analysis
```bash
# Calculate trending items
python3 analyzers/calculate_trending.py

# Analyze market volatility  
python3 analyzers/volatility_analyzer.py

# Run all analysis
python3 analyzers/run_analysis.py
```

### Maintenance Tasks
```bash
# Backup analytics tables
python3 maintenance/backup_tables.py

# Clean old data
python3 maintenance/cleanup_old_data.py

# Rotate logs
python3 maintenance/log_rotation.py
```

## Monitoring and Troubleshooting

### Log Files
Each script logs to its own file in the `logs/` directory:
- `logs/current_state.log` - Current state collector
- `logs/hourly_snapshot.log` - Snapshot collector
- `logs/notable_trades.log` - Notable trades collector
- `logs/health_check.log` - Health check results
- `logs/analysis.log` - Trend analysis results

### Common Issues

**1. No Recent Data**
```bash
# Check if collectors are running
crontab -l | grep market_analytics

# Check recent log activity
tail -f logs/cron.log

# Manual collection test
python3 collectors/current_state.py
```

**2. Database Connection Issues**
```bash
# Test database connectivity
python3 tests/test_quick.py

# Check credentials
cat .env

# Test direct connection
mysql -h $DB_HOST -u $DB_USER -p$DB_PASSWORD $DB_NAME -e "SELECT 1"
```

**3. Performance Issues**
```bash
# Check system resources
python3 tests/test_deployment.py

# Analyze query performance
python3 monitoring/performance_check.py
```

**4. Missing Indexes**
```bash
# Verify required indexes exist
python3 tests/test_essential.py
```

## Development

### Running Tests
```bash
# Development workflow
python3 tests/test_quick.py

# Pre-deployment validation
python3 tests/test_essential.py

# Production readiness
python3 tests/test_deployment.py
```

### Adding New Features
1. Add functionality to appropriate module
2. Add tests to `tests/` directory
3. Update documentation
4. Run comprehensive tests
5. Update cron schedule if needed

## Deployment

See [UBUNTU_DEPLOYMENT.md](UBUNTU_DEPLOYMENT.md) for detailed deployment instructions.

### Supported Platforms
- Ubuntu Server 18.04+
- CentOS/RHEL 7+
- Debian 10+
- Any Linux with Python 3.8+ and MySQL/MariaDB

### System Requirements
- **Memory**: 512MB minimum, 1GB recommended
- **Storage**: 5GB minimum for logs and data
- **CPU**: Single core sufficient for most RSC servers
- **Database**: MySQL 5.7+ or MariaDB 10.2+

## Security Notes

- Database user should have minimal required permissions
- Use dedicated analytics user account for deployment
- Regularly rotate database passwords
- Monitor log files for unusual activity
- Ensure `.env` file has secure permissions (600)

## Support

For issues or questions:
1. Check the troubleshooting section above
2. Review log files in `logs/` directory
3. Run the test suite to identify specific problems
4. Consult `docs/TESTING_GUIDE.md` for detailed testing procedures