# uMARKET System

## Overview

The uMARKET system is a specialized marketplace application for UPlanet that processes Nostr messages containing the `#market` tag and creates a dedicated web interface for displaying local market advertisements. The system now includes **global aggregation** capabilities that combine data from local UMAPs and swarm nodes to provide a comprehensive marketplace view.

## Architecture

### Components

1. **NOSTR.UMAP.refresh.sh** - Modified to detect `#market` tags and generate JSON advertisements with enhanced security
2. **\_uMARKET.generate.sh** - Creates the marketplace web interface with validation and error handling
3. **\_uMARKET.aggregate.sh** - **NEW**: Aggregates market data from local and swarm sources
4. **\_uMARKET.test.sh** - Comprehensive test script for the system
5. **\_uMARKET.test\_global.sh** - **NEW**: Test script for global aggregation
6. **\_uMARKET.monitor.sh** - Monitoring and health checking for the system

### Data Flow

```
Nostr Message with #market tag
           ↓
NOSTR.UMAP.refresh.sh processes message
           ↓
Validates and downloads images with security checks
           ↓
Creates validated JSON advertisement in ads/ directory
           ↓
_uMARKET.generate.sh validates and compiles all ads into market.json
           ↓
Generates secure web interface (_index.html)
           ↓
IPFS CID for the marketplace
           ↓
_uMARKET.aggregate.sh combines local + swarm data
           ↓
Global marketplace with complete UPlanet coverage
```

## Global Aggregation System

### Overview

The global aggregation system combines market data from:

* **Local UMAPs**: `~/.zen/tmp/$IPFSNODEID/UPLANET/__/_*/*/*/APP/uMARKET`
* **Swarm Nodes**: `~/.zen/tmp/swarm/*/UPLANET/__/_*/*/*/APP/uMARKET`

This provides a complete view of all market activity across the UPlanet network.

### Features

* ✅ **Multi-source aggregation**: Combines local and swarm data
* ✅ **Age filtering**: Configurable time-based filtering
* ✅ **Source tracking**: Each advertisement is tagged with its source
* ✅ **Enhanced interface**: Web interface shows source badges and statistics
* ✅ **Image consolidation**: Copies all associated images to a single location
* ✅ **Validation**: Filters out invalid or malformed advertisements
* ✅ **Statistics**: Real-time statistics on data sources and volumes

### Usage

```bash
# Basic global aggregation
./_uMARKET.aggregate.sh

# Local data only
./_uMARKET.aggregate.sh --local-only

# Swarm data only
./_uMARKET.aggregate.sh --swarm-only

# Recent advertisements only (24 hours)
./_uMARKET.aggregate.sh --max-age 24

# Custom output directory with verbose logging
./_uMARKET.aggregate.sh --output /path/to/output --verbose
```

### Output Structure

```
umarket_global/
├── ads/                    # Aggregated advertisements with source metadata
│   ├── message_id_1.json   # Enhanced with _source, _source_file, _aggregated_at
│   └── ...
├── Images/                 # Consolidated images from all sources
│   ├── UMAP_xxx_lat_lon_image1.jpg
│   └── ...
├── public/
│   └── market.json         # Compiled global advertisements
├── _index.html             # Enhanced web interface with source badges
└── index.html              # Redirect to IPFS CID
```

### Enhanced Advertisement Format

Each aggregated advertisement includes additional metadata:

```json
{
    "id": "message_id",
    "content": "Original message content",
    "author_pubkey": "nostr_public_key",
    "author_nprofile": "nostr:npub1...",
    "created_at": 1703000000,
    "location": {
        "lat": 48.85,
        "lon": 2.35
    },
    "local_images": ["image1.jpg", "image2.jpg"],
    "umap_id": "UMAP_xxx_lat_lon",
    "generated_at": 1703000000,
    "_source": "local|swarm",
    "_source_file": "/path/to/original/file.json",
    "_aggregated_at": 1703000000
}
```

## File Structure

### Local uMARKET

```
UMAP_APP/uMARKET/
├── ads/                    # JSON advertisement files
│   ├── message_id_1.json
│   ├── message_id_2.json
│   └── ...
├── Images/                 # Product images (validated)
│   ├── UMAP_xxx_lat_lon_image1.jpg
│   └── ...
├── public/                 # Generated by _uMARKET.generate.sh
│   └── market.json         # Compiled advertisements
├── _index.html             # Web interface
└── index.html              # Redirect to IPFS CID
```

### Global Aggregation

```
umarket_global/
├── ads/                    # Aggregated advertisements
├── Images/                 # Consolidated images
├── public/
│   └── market.json         # Global market data
├── _index.html             # Enhanced interface
└── index.html              # Redirect
```

## JSON Advertisement Format

Each advertisement is stored as a JSON file with the following structure:

```json
{
    "id": "message_id",
    "content": "Original message content with #market tag",
    "author_pubkey": "nostr_public_key",
    "author_nprofile": "nostr:npub1...",
    "created_at": 1703000000,
    "location": {
        "lat": 48.85,
        "lon": 2.35
    },
    "local_images": ["image1.jpg", "image2.jpg"],
    "umap_id": "UMAP_xxx_lat_lon",
    "generated_at": 1703000000
}
```

## Usage

### For Merchants

1. **Post a Nostr message** with the `#market` tag
2. **Include product images** as URLs in the message
3. **The system automatically**:
   * Validates and downloads images with security checks
   * Creates a validated JSON advertisement
   * Includes it in the marketplace interface
   * Makes it available for global aggregation

Example message:

```
🍎 Fresh organic apples from my garden! 
Perfect for pies and cider. 
Price: 1 DU/kg
#market #organic #local
https://example.com/apples.jpg
```

### For Consumers

1. **Visit the local marketplace** at the UMAP's IPFS URL
2. **Visit the global marketplace** using the aggregation system
3. **Browse products** by category, price, or location
4. **Filter by source** (local vs swarm)
5. **Contact sellers** via their Nostr profile
6. **View product images** and details

## Features

### Enhanced Security

* ✅ URL validation for image downloads
* ✅ File type validation for images
* ✅ Size limits for uploaded images (configurable)
* ✅ Filename sanitization to prevent path traversal
* ✅ JSON validation before processing
* ✅ Timeout and retry limits for downloads

### Automatic Processing

* ✅ Detects `#market` tags in Nostr messages
* ✅ Downloads and validates product images
* ✅ Creates structured JSON advertisements
* ✅ Generates responsive web interface
* ✅ **Global aggregation of all market data**

### Web Interface

* ✅ Modern, mobile-friendly design
* ✅ Product cards with images
* ✅ Search and filter functionality
* ✅ Author information and contact details
* ✅ Location-based organization
* ✅ **Source badges (local/swarm)**
* ✅ **Real-time statistics**
* ✅ **Enhanced filtering options**

### Data Management

* ✅ Automatic cleanup of old advertisements (6 months)
* ✅ Notification to authors when ads expire
* ✅ IPFS-based storage and distribution
* ✅ JSON-based data structure for easy processing
* ✅ **Global data consolidation**
* ✅ **Age-based filtering**

### Validation & Error Handling

* ✅ JSON schema validation
* ✅ Image file validation
* ✅ Comprehensive error reporting
* ✅ Graceful failure handling
* ✅ **Multi-source validation**

## Testing

### Local uMARKET Testing

```bash
cd Astroport.ONE/tools/
./_uMARKET.test.sh
```

### Global Aggregation Testing

```bash
cd Astroport.ONE/tools/
./_uMARKET.test_global.sh
```

This comprehensive test will:

1. Create mock local and swarm data
2. Test local-only aggregation
3. Test swarm-only aggregation
4. Test full aggregation (local + swarm)
5. Test age filtering
6. Verify enhanced web interface
7. Verify image copying
8. Validate source distribution

### Test Scenarios

* **Local Generation**: Standard marketplace creation
* **JSON Validation**: Detection of malformed JSON files
* **Logging**: Detailed output for debugging
* **Size Limits**: Image size validation
* **File Structure**: Verification of generated files
* **HTML Content**: Validation of web interface
* **Global Aggregation**: Multi-source data combination
* **Source Tracking**: Verification of data provenance
* **Age Filtering**: Time-based data filtering

## Advanced Usage

### \_uMARKET.generate.sh Options

```bash
# Basic usage
./_uMARKET.generate.sh /path/to/market/data

# With validation
./_uMARKET.generate.sh --validate /path/to/market/data

# With logging and size limits
./_uMARKET.generate.sh --log --max-size 5 /path/to/market/data

# Full validation with custom settings
./_uMARKET.generate.sh --validate --max-size 10 --log /path/to/market/data
```

### \_uMARKET.aggregate.sh Options

```bash
# Basic global aggregation
./_uMARKET.aggregate.sh

# Local data only
./_uMARKET.aggregate.sh --local-only

# Swarm data only
./_uMARKET.aggregate.sh --swarm-only

# Recent advertisements only (24 hours)
./_uMARKET.aggregate.sh --max-age 24

# Custom output with verbose logging
./_uMARKET.aggregate.sh --output /custom/path --verbose
```

### Configuration Options

#### \_uMARKET.generate.sh

* `--validate`: Enable JSON validation
* `--max-size`: Set maximum image size in MB (default: 10)
* `--log`: Enable detailed logging
* `-h, --help`: Show help information

#### \_uMARKET.aggregate.sh

* `--output`: Set output directory (default: /home/fred/.zen/flashmem/umarket\_global)
* `--max-age`: Set maximum advertisement age in hours (default: 168)
* `--local-only`: Process only local data
* `--swarm-only`: Process only swarm data
* `--verbose`: Enable detailed logging
* `-h, --help`: Show help information

## Integration

The uMARKET system integrates seamlessly with the existing UPlanet infrastructure:

* **UMAPs** automatically detect and process market messages
* **Sectors** aggregate market activity from multiple UMAPs
* **Regions** provide broader market overview
* **IPFS** ensures decentralized storage and access
* **Swarm** provides global data aggregation
* **Global Aggregation** combines all market data for complete coverage

## Security Considerations

### Image Processing

* Maximum file size limits (configurable)
* File type validation (jpg, jpeg, png, gif only)
* URL format validation
* Download timeout and retry limits
* Filename sanitization

### Data Validation

* JSON schema validation
* Required field checking
* Content escaping for XSS prevention
* Author verification
* Multi-source data validation

### Network Security

* Download rate limiting
* Maximum redirect limits
* Timeout protection
* Error handling for network failures

### Global Aggregation Security

* Source tracking for data provenance
* Age-based filtering to prevent stale data
* Validation of aggregated data
* Secure handling of multi-source content

## Performance Optimizations

* Lazy loading of images in web interface
* Efficient JSON compilation
* Minimal IPFS operations
* Responsive web design
* Caching of validated files
* **Parallel processing of multiple sources**
* **Incremental aggregation updates**

## Monitoring

### System Health Monitoring

```bash
cd Astroport.ONE/tools/
./_uMARKET.monitor.sh
```

### Health Report Generation

```bash
./_uMARKET.monitor.sh --report
```

The monitoring system provides:

* Dependency checking
* IPFS node status
* Data integrity validation
* Performance metrics
* Error rate monitoring
* Resource usage tracking

## Future Enhancements

Potential improvements for the uMARKET system:

* [ ] Price extraction and sorting
* [ ] Category-based filtering
* [ ] Distance-based search
* [ ] Seller ratings and reviews
* [ ] Direct messaging between buyers and sellers
* [ ] Payment integration
* [ ] Inventory management
* [ ] Market analytics and trends
* [ ] Real-time notifications
* [ ] Multi-language support
* [ ] Advanced image processing (thumbnails, compression)
* [ ] Blockchain integration for escrow services
* [ ] **Real-time global aggregation updates**
* [ ] **Cross-node data synchronization**
* [ ] **Advanced source reliability scoring**
* [ ] **Geographic clustering and analysis**

## Technical Details

### Dependencies

* `jq` - JSON processing and validation
* `ipfs` - Decentralized storage
* `nostr_send_note.py` - Nostr communication
* `wget` - Secure image downloading
* `file` - File type detection

### Error Handling

* Graceful degradation on network failures
* Detailed error logging
* Validation feedback
* Automatic cleanup of invalid files
* Multi-source error isolation

### Monitoring

* Download success/failure tracking
* File validation statistics
* Performance metrics
* Error rate monitoring
* Global aggregation statistics
* Source reliability metrics


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://astroport-1.gitbook.io/astroport.one/tools/_umarket.readme.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.