Tuesday, October 20, 2015

Getting Oracle Linux on Amazon EC2

If you feel the need to roll your own Oracle Linux install on Amazon EC2 (since Oracle no longer provides an officially-supported AMI, and you may not be too keen on using one of the community AMI's):

(1) launch an EC2 image using the CentOS 6.5 AMI from the Marketplace (which is free..) then log in as root (not ec2-user)

(2) import the Oracle GPG key
1
2
cd /etc/pki/rpm-gpg/
curl -O https://oss.oracle.com/ol6/RPM-GPG-KEY-oracle

(3) use the Oracle specific rules to convert the CentOS 6 to OL from this documentSpecifically, run the following commands as root:
1
2
curl -O https://linux.oracle.com/switch/centos2ol.sh
sh centos2ol.sh

(4) Synchronize the yum repository
1
yum -y upgrade

Once the command completes, the end-user should have a fully-patched Oracle Linux 6.7.

If you intend to install Oracle Database 11gR2, you can apply all the necessary packages and kernel parameters with this command:
1
yum install oracle-rdbms-server-11gR2-preinstall -y

and if you will install Oracle Database 12c R1, use the following:
1
yum install oracle-rdbms-server-12cR1-preinstall -y

Tuesday, October 06, 2015

Collecting Haze Data with Arduino, Raspberry Pi, and Amazon DynamoDB

I finally decided to put those Sharp airborne dust sensors which I bought during the last haze epidemic (in 2013) to good use, along with the small mountain of parts I have at home.

I used this very simple Arduino circuit and code snippet to read the Sharp sensor.  I ended up using a 100uF capacitor and 220-ohm resistor because those are what I had on hand; performance seems unimpaired. I used an Arduino Uno, because the dust sensor needs 5V. The sensor is very noisy and jumpy, so I used an interquartile mean and 200 measurements (discarding the bottom 25% and top 25%) to get a more robust reading. The raw value still jumps around by 1 LSB.


I also modified the slope function so that a full-range reading is 250 ug/m^3 (the original formula is here). Here is the modified Arduino code:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
// program to read a Sharp GP2Y1010AU0F dust sensor
// this sensor produces a dust value with a maximum value of 250 ug/m3

#define NUMSAMPLES 200
#define dustPin 0
#define ledPower 2
#define delayTime 280
#define delayTime2 40
#define offTime 9680

void setup() {
  Serial.begin(9600);
  pinMode(ledPower, OUTPUT);
  pinMode(4, OUTPUT);
}

// swap sort algorithm
void swapsort(int *sorted, int num) {
  boolean done = false;    // flag to know when we're done sorting              
  int j = 0;
  int temp = 0;

  while(!done) {           // simple swap sort, sorts numbers from lowest to highest
    done = true;
    for (j = 0; j < (num - 1); j++) {
      if (sorted[j] > sorted[j + 1]){     // numbers are out of order - swap
        temp = sorted[j + 1];
        sorted [j+1] =  sorted[j] ;
        sorted [j] = temp;
        done = false;
      }
    }
  }
}


// read the dust sensor, implementing an interquartile mean
// thanks to STMicro Application Note 3964 "How to design a simple temperature measurement application using the STM32L-DISCOVERY"
int readRawDustValue() {
  int i = 0;
  int rawVal[NUMSAMPLES];

  for (i = 0; i < NUMSAMPLES; i++) {
    // ledPower is any digital pin on the arduino connected to Pin 3 on the sensor
    digitalWrite(ledPower, LOW); // power on the LED
    delayMicroseconds(delayTime);

    rawVal[i] = analogRead(dustPin); // read the dust value via pin 5 on the sensor
    delayMicroseconds(delayTime2);

    digitalWrite(ledPower, HIGH); // turn the LED off
    delayMicroseconds(offTime);
  }

  // now we have the raw values, sort them
  swapsort(rawVal, NUMSAMPLES);
  
  // drop the lowest 25% and highest 25% of the readings
  long dustVal = 0;
  for (i = NUMSAMPLES / 4; i < (NUMSAMPLES * 3 / 4); i++) {
    dustVal += rawVal[i];
  }
  dustVal /= (NUMSAMPLES / 2);
  return (dustVal);  
}


// convert the raw count to a dust value
// the full-range signal is 771 counts = 3.76V
// based on http://www.howmuchsnow.com/arduino/airquality/
//
// dust density (mg/m3) = 0.172 * V - 0.1

float calcDustDensity (int rawVal) {
  float calcVoltage = rawVal * (5.0 / 1024);
  
  // I use a different figure so that 771 counts = 248 ug/m3
  float dustDensity = ((calcVoltage * 0.688) - 0.1) * 100;
  
  if (dustDensity < 0) dustDensity = 0;
  return (dustDensity);
}


void loop() {
  int dustValue = readRawDustValue();
  float dustDensity = calcDustDensity(dustValue);
  
  Serial.print("Raw Dust Value = ");
  Serial.println(dustValue);
  Serial.print("Dust Density = ");
  Serial.println(dustDensity);
}

Unfortunately the Arduino Uno only has 2K of RAM and mine did not have an Ethernet or Wi-Fi shield, so I decided to use a Raspberry Pi (Model B, the old single-core one) to read the Arduino Uno and talk to DynamoDB.

Basically, the Arduino appears as a serial port to the RasPi and I read the values using pySerial and uploaded them using boto.

Surprisingly the RasPi behaves exactly like a "real" Linux box, installing the AWS CLI and Python SDK (boto3 and boto) is exactly the same as on a large machine. There were no hiccups during the installation (albeit the installation took a long time).


The Python script is as follows (my first Python program!) note the hard-coded AWS credentials which is really a terrible practice.


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
#!/usr/bin/python

import boto
from boto import dynamodb2
from boto.dynamodb2.table import Table

import datetime
import time
import serial
import re
import sys

# monkey hacking to work around "inexact numeric" issue in boto
import decimal
from boto.dynamodb.types import DYNAMODB_CONTEXT
# Inhibit Inexact Exceptions
DYNAMODB_CONTEXT.traps[decimal.Inexact] = 0
# Inhibit Rounded Exceptions
DYNAMODB_CONTEXT.traps[decimal.Rounded] = 0


conn = dynamodb2.connect_to_region(
 'ap-southeast-1',
 aws_access_key_id='AKIAxxxx',
 aws_secret_access_key='R0KIxxxx',
)
table = Table(
 'dustValues',
 connection=conn
)

# open the serial port (we need to be setuid root for this)
serialport = serial.Serial("/dev/ttyACM0", 9600, timeout=0.5)

rawDustValue = 0
dustDensity = 0

oldDustValue = 0
oldDustDensity = 0

# to put a blank line..
print "\n"

while True:
 command = serialport.readline()
 matchObj = re.match( '^(.*) = (.*)$', command, re.M | re.I)
 if (matchObj):
  # hash is unixTimestamp
  unixTimestamp = int(time.time())
  timestamp = time.strftime("%Y%m%d%H%M%S")

  # Var =  Raw Dust Value
  # Val =  137
  # Var =  Dust Density
  # Val =  81.51
  if (matchObj.group(1) == 'Raw Dust Value'):
   rawDustValue = int(matchObj.group(2))
   # print "raw dust value = ", rawDustValue

  if (matchObj.group(1) == 'Dust Density'):
   dustDensity = int(float(matchObj.group(2)))
   # print "dust density = ", dustDensity

 if ((rawDustValue != oldDustValue) and (dustDensity != oldDustDensity)):
  oldDustValue = rawDustValue
  oldDustDensity = dustDensity

  # print "hash = ", str(unixTimestamp)

  # calculate the PSI (this is a very approximate value)
  # based on dustDensity (in ug/m3) and this
  # http://www.haze.gov.sg/docs/default-source/faq/computation-of-the-pollutant-standards-index-(psi).pdf
  # we only use the 24-hour PM2.5
  # note that the Sharp sensor can't distinguish particle size
  # so PM10 particles are also falsely measured

  psi = 0
  if (dustDensity <= 12):
   psi = dustDensity * 4.17
  elif (dustDensity > 12 and dustDensity <= 55):
   psi = 51 + ((dustDensity - 13) * 1.17)
  elif (dustDensity > 55 and dustDensity <= 150):
   psi = 101 + ((dustDensity - 56) * 1.05)
  elif (dustDensity > 150):
   psi = 201 + (dustDensity - 105)

  psi = int(psi)

  # shorten timestamp so it fits on the tiny PiTFT screen
  ts = int(unixTimestamp) - 1444116000

  print ts, ":",
  print "raw=", rawDustValue,
  print " ug/m3=", dustDensity,
  print " PSI=", psi, "   \r",
  sys.stdout.flush()

  # write to the table
  try:
   table.put_item(data={
    'unixTimestamp': int(unixTimestamp),
    'timestamp': timestamp,
    'rawDustValue' : int(rawDustValue),
    'dustDensity' : int(dustDensity),
    'psi' : int(psi)
   })
  except:
   # do nothing
   pass

I then placed the Python script in /etc/rc.local (making sure to append an ampersand so that booting would complete).

And voila: