Friday, October 24, 2025

An Example of Kafka Topic Consume and Produce by AWS Lambda in Go

In this article, I demonstrate how to consume messages from Amazon MSK (Managed Streaming for Apache Kafka) with AWS Lambda and then produce messages using Lambda in Go.

For security best practices, it’s recommended to use IAM authentication with SASL when sending messages to MSK.

Currently, there are only a few Go libraries available for interacting with MSK. Some of them rely on C dependencies, which is not ideal for Go developers, as it requires enabling CGO and managing additional language dependencies.

To avoid these issues, I used the segmentio/kafka-go library, which is a Kafka client implemented entirely in Go.

MSK IAM SASL Implementation Example

func NewProducer() (*Producer, error) {
	kafkaBrokers := os.Getenv("KAFKA_BROKERS")
	if kafkaBrokers == "" {
		return nil, fmt.Errorf("KAFKA_BROKERS environment variable is required")
	}

	// Validate that we have at least one broker
	brokers := strings.Split(kafkaBrokers, ",")
	if len(brokers) == 0 {
		return nil, fmt.Errorf("KAFKA_BROKERS must contain at least one broker")
	}

	// Trim whitespace from broker addresses
	for i, broker := range brokers {
		brokers[i] = strings.TrimSpace(broker)
		if brokers[i] == "" {
			return nil, fmt.Errorf("invalid empty broker address in KAFKA_BROKERS")
		}
	}

	log.Printf("Kafka producer initializing with brokers: %v", brokers)

	awsCfg, err := config.LoadDefaultConfig(context.Background())
	if err != nil {
		return nil, fmt.Errorf("failed to load AWS config: %w", err)
	}

	// Create IAM SASL mechanism with credentials provider (not static credentials)
	iamSaslMechanism := &aws_msk_iam_v2.Mechanism{
		Signer:      signer.NewSigner(),
		Credentials: awsCfg.Credentials,
		Region:      awsCfg.Region,
	}

	// Configure transport with TLS and SASL
	sharedTransport := &kafka.Transport{
		SASL: iamSaslMechanism,
		TLS:  &tls.Config{},
	}

	// Initialize Kafka writer immediately for provisioned Lambda
	kafkaWriter := &kafka.Writer{
		Addr:         kafka.TCP(brokers...),
		RequiredAcks: kafka.RequireOne,
		BatchTimeout: 10 * time.Millisecond, // For Low latency
		BatchSize:    1,                     // Send messages immediately
		Compression:  kafka.Snappy,          // Use Snappy compression
		Transport:    sharedTransport,
	}

	log.Printf("Kafka producer initialized successfully with %d brokers", len(brokers))

	return &Producer{
		kafkaWriter: kafkaWriter,
	}, nil
}
There is a Go Test, you can put a break point and follow the process how it works. you may need a local stack or AWS Mock to test fully

func TestHandlerWithJSONFile(t *testing.T) {

	// Set up test environment (optional)
	os.Setenv("KAFKA_BROKERS", "localhost:9098,localhost:9098")

	// Read the JSON file
	jsonData, err := os.ReadFile("test/kafka-event-id-41350679.json")
	if err != nil {
		t.Fatalf("Failed to read test JSON file: %v", err)
	}

	// Unmarshal into KafkaEvent
	var event events.KafkaEvent
	if err := json.Unmarshal(jsonData, &event); err != nil {
		t.Fatalf("Failed to unmarshal JSON into KafkaEvent: %v", err)
	}

	// Call the handler function
	ctx := context.Background()
	err = handler(ctx, event)

	// Check result
	if err != nil {
		t.Errorf("Handler returned error: %v", err)
	} else {
		t.Log("Handler executed successfully")
	}

	// Log some details about what was processed
	for topic, records := range event.Records {
		t.Logf("Processed topic: %s with %d records", topic, len(records))
		for _, record := range records {
			t.Logf("  - Partition: %d, Offset: %d", record.Partition, record.Offset)
		}
	}
}
For a whole implementation example, you can refer to the pull request or the branch linked below, which show exactly how to set this up.

https://github.com/gogo-boot/cdk-lambda-go/pull/1



A Practical Guide to Deploying Go Lambdas with AWS CDK

I needed to create a Lambda function with low latency and fast computation. To achieve this, I chose Go, as it is easy to learn and consumes fewer resources compared to other languages.

However, I found that there are very few examples available for building Lambda functions in Go, especially when using AWS CDK or SAM for deployment. Even AI tools couldn't provide sufficient guidance, likely because there isn’t much related content available online.

As a result, I decided to build the solution myself.

Below is a simple example, It is deployed using AWS CDK and utilizes an AWS Linux image, which is lightweight. The CDK makes it straightforward to deploy and manage the Lambda function end to end.

import * as cdk from 'aws-cdk-lib';
import {Duration, RemovalPolicy} from "aws-cdk-lib";
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as path from "path";


export interface MyLambdaStackProps extends cdk.StackProps {}

export class MyLambdaStack extends cdk.Stack {

    constructor(scope: Construct, id: string, props: MyLambdaStackProps) {
        super(scope, id, props);

        // Define the ExampleLambda Lambda function
        const goLambda = new lambda.Function(this, "ExampleLambda", {
            runtime: lambda.Runtime.PROVIDED_AL2023, // Use the custom runtime
            handler: "bootstrap", // Go binary name
            architecture: lambda.Architecture.ARM_64, // Use ARM architecture, cheaper and better performance

            currentVersionOptions: {
                removalPolicy: RemovalPolicy.RETAIN
            },

            // lambda.Code.fromAsset will pack artifact from "/asset-output" directory.
            // therefor the "bootstrap" file must be located in this directory.
            code: lambda.Code.fromAsset(path.join(__dirname, '../example-lambda'), {  // Path to Go binary
                bundling: {
                    image: cdk.DockerImage.fromRegistry("golang:1.25"),
                    command: [
                        'bash', '-c',
                        'GOARCH=arm64 GOOS=linux CGO_ENABLED=0 go build -tags lambda.norpc -o /asset-output/bootstrap main.go'
                    ],
                },
            }),
            // environment: {
            //     'DB_HOST': '',
            // },
            // vpc: this.vpc,
            // vpcSubnets: {
            //     subnets: [subnet1, subnet2, subnet3]
            // },
            // securityGroups: [MySecurityGroup],
            // timeout: cdk.Duration.seconds(10) // Limit timeout to 10 seconds
        });
    }
}


you can have full example code https://github.com/gogo-boot/cdk-lambda-go It has better description in README.md.

You may put the Eventsource `goLambda.addEventSource` for triggering the Lambda by event.

also Provision the Lambda and Auto Scale by `goLambda.addAutoScaling({ minCapacity: 1, maxCapacity: 5 });`, so you lambda will be pre provisioned and run without cold start. so it responses very fast.

Thursday, November 4, 2021

AWS Secret Manager sample code with golang aws-sdk-go-v2 SDK

AWS Secret Manager provides sample code. It is very good to use for quick start. 

But, I see often, they provide sample code with old version of SDK. 

In this time, I program with Golang. and the aws-sdk-go is old version sdk. 

As soon as I pushe the code, I know, it will be alerted by source scanning software. 

I had to modify the sample code. as below

 

package aws

// Use this code snippet in your app.
// If you need more information about configurations or implementing the sample code, visit the AWS docs:
// https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/setting-up.html

import (
	"context"
	"encoding/base64"
	"errors"
	"fmt"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
	"github.com/aws/smithy-go"
)

func getSecret() {
	secretName := "arn:aws:secretsmanager:eu-central-1:.... put your secretmanager ARN"
	region := "eu-central-1"

	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion(region),
	)
	if err != nil {
		// handle error
	}

	//Create a Secrets Manager client
	svc := secretsmanager.NewFromConfig(cfg)
	input := &secretsmanager.GetSecretValueInput{
		SecretId:     aws.String(secretName),
		VersionStage: aws.String("AWSCURRENT"), // VersionStage defaults to AWSCURRENT if unspecified
	}

	// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
	// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html

	result, err := svc.GetSecretValue(context.TODO(), input)
	if err != nil {

		var apiErr smithy.APIError
		if errors.As(err, &apiErr) {
			code := apiErr.ErrorCode()
			message := apiErr.ErrorMessage()
			// handle error code
			fmt.Println("error code: " + code + " message : " + message)
			return
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return
	}

	// Decrypts secret using the associated KMS CMK.
	// Depending on whether the secret is a string or binary, one of these fields will be populated.
	var secretString, decodedBinarySecret string
	if result.SecretString != nil {
		secretString = *result.SecretString
	} else {
		decodedBinarySecretBytes := make([]byte, base64.StdEncoding.DecodedLen(len(result.SecretBinary)))
		len, err := base64.StdEncoding.Decode(decodedBinarySecretBytes, result.SecretBinary)
		if err != nil {
			fmt.Println("Base64 Decode Error:", err)
			return
		}
		decodedBinarySecret = string(decodedBinarySecretBytes[:len])
	}

	// Your code goes here.
	fmt.Println(secretString)
	fmt.Println(decodedBinarySecret)
}

Friday, December 21, 2018

maven with NTLM Proxy Server

While I am migrating my project to Java 11, I was facing many issues. I was not sure if the 3rd Party Framesworks work fine. I had a doubt. So I tested several Frameworks. So far, Eclipslink and Jmokit was not running fine on Java11. I have reported to Jmokit. Eclipselink has apparently fixed something in the latest code, but the latest version is not released yet. It is laid in Snapshot repository. I am using Company Nexus Repository which I am not allow to change the configuration for adding Snapshot Repository. To access the Repository, I had to configure Proxy in maven setting.xml. But, I was not able to access the Repository yet. I realized the Proxy Server is required NTML based Authentication. Maven doesn't support NTLM Proxy Authentication. It required another way to solve.

I found out that either use CNTLM or wagon-http-lightweight. CNTLM would be good for various reson. If you use several application behind Proxy, like a VirtualBox, Git, Yum, SVN, ... But, I just need for a maven. so I have chosen wagon-http-lightweight.

It is fairly easy to use this. You can download the file from https://mvnrepository.com/artifact/org.apache.maven.wagon/wagon-http-lightweight/2.2 This is just 15 Kbytes. locate the dowonloaded file under M2_HOME/lib/ext
Now, You need to configure the maven xml files. You can use the password as plaintext in the configuration. But, I recommend to use encrypted password for the security. maven will recognize automatically when the password starts with "{" and will decrypt it before authenticate with Proxy Server. I will create a master password and user password by maven. masterpassword will be used for decrypting userpassword with in maven. userpassword is your NTLM userpassword

USER_HOME/.m2/settings-security.xml
mvn -emp masterpassword

{SPg1nt21S2MHuw0Hy8MJaEF7Gc7dK25UWGDYKHupNCw=}
<settingsSecurity>
  <master>{SPg1nt21S2MHuw0Hy8MJaEF7Gc7dK25UWGDYKHupNCw=}</master>
</settingsSecurity>
 


Set up your proxy in USER_HOME/.m2/setting.xml

mvn -ep userpassword
{7ut6v4FFiJMHtwsmYrsmLMcPoDBGmbz/kgcQ6Vks+/0=}

<proxies>
    <proxy>
   <id>internet-proxy</id>
   <active>true</active>
   <protocol>http</protocol>
   <host>###ProxyHost###</host>
   <username>###Username###</username>
   <password>{7ut6v4FFiJMHtwsmYrsmLMcPoDBGmbz/kgcQ6Vks+/0=}</password>
   <port>###Proxy Port###</port>
   <nonProxyHosts>localhost|127.0.0.1</nonProxyHosts>
    </proxy>
 </proxies>  


PS. Security is good. but it makes develpers crazy. I hope there is a simple way to secure and easy to devlop without configure extra. it takes so much time to configure and get it.

Thursday, August 10, 2017

Curl Command example with client Key to https Server

To communicate with https Server by curl command, if a client private key is required, it is not simple like as i thought. my_client.12 is my client key and I need to convert to the other format. Before starting, check your key.
openssl pkcs12 -info -in my_client.12



Check the Server sertificate
openssl s_client -showcerts -connect www.domain.com:443


If you see that all right. you will be able to communicate with the https server which requires client key.

Convert the client key
openssl pkcs12 -in my_client.12 -out client.pem -clcerts -nokeys
openssl pkcs12 -in my_client.12 -out key.pem -nocerts
openssl rsa -in key.pem -out newkey.pem


And now finally, you can communicate with the server.
curl -v -G  -k --key newkey.pem --cert ./client.pem:password https://www.domain.com/path

Download Files from AWS S3 Bucket via SQS message



Our Infrastructure put a log file in S3 bucket. I need to get only this new Log file from the S3 bucket and parse it put into DB. The Log parse Servers are clustered which might be inturruped while downloading and might download again which another cluster server is currently parsing the log file.
There is a solution. When new files are created, the S3 Bucket will send a message to a Simple Queue Service(SQS). The Log Parsing Servers will pull the messages every minute. If a message already pulled, SQS will hide a message in determined period until the delete action. The delete action will be called from parsing server after downloading.
To run the code, you will need a python3 or higher and boto3(Amazon Web Services (AWS) SDK for Python)
yum install epel-release
yum install python34-pip
pip3 install boto3



There is a python3 code sample.
import boto3
import json
import traceback
import os
import logging, sys
import tarfile

#logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

# Get the service resource
sqs = boto3.resource('sqs', region_name='eu-central-1')
s3_client = boto3.client('s3', region_name='eu-central-1')

# Get the queue
queue = sqs.get_queue_by_name(QueueName='Your SQS Queue Name')

for message in queue.receive_messages(MaxNumberOfMessages=10):
  try:
    if message is not None:
      # Parsing event message from s3 bucket
      s3 = json.loads(message.body)['Records'][0]['s3']
      bucket = s3['bucket']['name']
      key = s3['object']['key']
      
      logging.debug('bucket :'+bucket)
      logging.debug('key :'+key)
      
      # Get filename and directory from key
      filename = os.path.basename(key)
      directory = '/your_prefix_dir/' + os.path.dirname(key)
      
      logging.debug('filename :'+filename)
      logging.debug('directory :'+directory)
      
      # Create Directory if it is not exist
      if not os.path.exists(directory):
          os.makedirs(directory)  
                          
      # Download Log File
      s3_client.download_file(bucket, key, directory + filename)
      logging.debug('Download completed')
      
      # Extract tar.gz File
      tar = tarfile.open(directory + filename, "r:gz")
      tar.extractall(directory)
      tar.close()  
      logging.debug('extract file completed')
         
      # Remove tar.gz File
      os.remove(directory + filename)
      
      # Remove SQS message    
      message.delete()
      
  except ValueError:
    # SQS is dedicated for S3 event message. If there is wrong message from other service, leave message body and remove the message
    logging.error('Message format is not valid. Delete message :' + message.body)
    message.delete()
  except Exception:
    logging.error(traceback.format_exc()) 
  else:
    logging.info('finish')



If you see message from the S3 bucket like below command. It is ready to pull the log file by the python script. Watch out, the SQS hide the message default 20 second. it won't be visiable.
aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/your account number/queueName --max-number-of-messages 1 --region eu-central-1

you can excute the python script like below 20 seconds later. flock is for avoiding concurrent excution.
flock -n /home/your Dir/lock/s3-copy.lock -c "/usr/bin/python3 /your_prefix_dir/s3-copy.py"

The log file are downloaded and the tar.gz files are extracted.

Thursday, August 3, 2017

Generate Unique Code for Coupons or Vouchers in java

Please see the Maven Dependency
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>commons-lang</groupId>
   <artifactId>commons-lang</artifactId>
  </dependency>



import java.math.BigInteger;

public class Coupon {

    private byte numberOfChar;

    private BigInteger code;

    private BigInteger crc;

    public byte getNumberOfChar() {
        return numberOfChar;
    }

    public void setNumberOfChar(byte numberOfChar) {
        this.numberOfChar = numberOfChar;
    }

    public BigInteger getCode() {
        return code;
    }

    public void setCode(BigInteger code) {
        this.code = code;
    }

    public BigInteger getCrc() {
        return crc;
    }

    public void setCrc(BigInteger crc) {
        this.crc = crc;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof Coupon)) {
            return false;
        }

        Coupon c = (Coupon) o;
        if (code.equals(c.code)) {
            return true;
        }
        return false;

    }

    @Override
    public int hashCode() {
        int hash = 5;
        hash = 89 * hash + (code != null ? code.hashCode() : 0);
        hash = 89 * hash + (crc != null ? crc.hashCode() : 0);
        return hash;
    }

}




public class CouponGenerateError extends Exception {

    public CouponGenerateError(String message) {
        super(message);
    }

    private static final long serialVersionUID = -8232285755941178115L;

}




import java.math.BigInteger;
import java.security.SecureRandom;
import java.util.LinkedHashSet;
import java.util.Set;

public class CouponGenerator {

    private SecureRandom random = new SecureRandom();

    /**
     * Represent of number of CRC chracter. If it is 1. One Character of CRC code will be generated and attached next to
     * coupon code
     */
    static int CRC_SIZE = 1;

    /**
     * CRC Generator should never be changed unless you like to invalidate all previously published coupons. It is a key
     * value to prove if the CRC value is right or not
     */
    static int CRC_GENERATOR = 31;

    /**
     * @param numberOfChar
     *            is define how long is the Voucher code. Voucher code consists of coupon code and CRC code
     * @param numberOfCoupon
     *            is define how many Vouchers must be generated
     * @return
     * @throws CouponGenerateError
     */
    public Set getCouponSet(byte numberOfChar, long numberOfCoupon) throws CouponGenerateError {

        isValidNumberOfChar(numberOfChar);

        Set couponSet = new LinkedHashSet();

        while (numberOfCoupon > couponSet.size()) {
            couponSet.add(generateCoupon(numberOfChar));
        }

        return couponSet;
    }

    /**
     * @param numberOfChar
     *            It must be bigger than 2 and smaller than 12. If the parameter is 2, it will have 32 cases of coupon
     *            code. It is meaning less to generate. 
* If the parameter is bigger than 12. Internal long type variable can not calculate it. It will make a * byte overflow and return garbage value. Because, Long is 8 byte which is 64 bits. Each coupon chracter * takes 5 bit. 13 Chracters will take 65 bits * @throws CouponGenerateError */ private void isValidNumberOfChar(byte numberOfChar) throws CouponGenerateError { if (numberOfChar < 3 || numberOfChar > 12) { throw new CouponGenerateError( "Invalid numberOfChar for Coupon chracters. It must be bigger than 2 and smaller than 12"); } } /** * @param numberOfChar * length of Alphanumeric code value is defined by numberOfChar * @param numberOfChar * must be bigger than 3 or equal and smaller than 12 or equal chracters * @return * @throws CouponGenerateError */ public Coupon getCoupon(byte numberOfChar) throws CouponGenerateError { isValidNumberOfChar(numberOfChar); return generateCoupon(numberOfChar); } private Coupon generateCoupon(byte numberOfChar) throws CouponGenerateError { Coupon coupon = new Coupon(); coupon.setNumberOfChar(numberOfChar); // Create 5 random bits per character. 5 bits represent Base32 Encoding coupon.setCode(new BigInteger((numberOfChar - CRC_SIZE) * 5, random)); coupon.setCrc(CouponUtil.calculateCrc(coupon)); return coupon; } }



import java.math.BigInteger;

import org.apache.commons.lang.StringUtils;

public class CouponUtil {

    /**
     * @param coupon
     *            
     * 7.  Base 32 Encoding with Extended Hex Alphabet
     * 
     *    The following description of base 32 is derived from [7].  This
     *    encoding may be referred to as "base32hex".  This encoding should not
     *    be regarded as the same as the "base32" encoding and should not be
     *    referred to as only "base32".  This encoding is used by, e.g.,
     *    NextSECure3 (NSEC3) [10].
     * 
     *    One property with this alphabet, which the base64 and base32
     *    alphabets lack, is that encoded data maintains its sort order when
     *    the encoded data is compared bit-wise.
     * 
     *    This encoding is identical to the previous one, except for the
     *    alphabet.  The new alphabet is found in Table 4.
     * 
     *                  Table 4: The "Extended Hex" Base 32 Alphabet
     * 
     *          Value Encoding  Value Encoding  Value Encoding  Value Encoding
     *              0 0             9 9            18 I            27 R
     *              1 1            10 A            19 J            28 S
     *              2 2            11 B            20 K            29 T
     *              3 3            12 C            21 L            30 U
     *              4 4            13 D            22 M            31 V
     *              5 5            14 E            23 N
     *              6 6            15 F            24 O         (pad) =
     *              7 7            16 G            25 P
     *              8 8            17 H            26 Q
     * 
     * @return
     */
    public static String getVoucherString(Coupon coupon) {

        byte numberOfChar = coupon.getNumberOfChar();

        String code = coupon.getCode().toString(32);

        if (!isCodeSizeRight(numberOfChar, code)) {
            code = leftPadding0(numberOfChar, code);
        }
        // Debug purpose
        // System.out.println("code : " + code);
        // System.out.println("bit code : " + coupon.getCode().toString(2));
        // System.out.println("CRC : " + coupon.getCrc().toString(32));
        return code + coupon.getCrc().toString(32);
    }

    private static String leftPadding0(byte numberOfChar, String code) {

        return StringUtils.repeat("0", numberOfChar - CouponGenerator.CRC_SIZE).substring(
                code.length() % numberOfChar)
                + code;

    }

    private static boolean isCodeSizeRight(byte numberOfChar, String code) {
        return code.length() % (numberOfChar - CouponGenerator.CRC_SIZE) == 0;
    }

    /**
     * It validate the CRC value from the coupon.
     * 
     * @param coupon
     * @return
     */
    static boolean isValidCrc(Coupon coupon) {

        if (coupon.getCrc().equals(calculateCrc(coupon))) {
            return true;
        } else {
            return false;
        }

    }

    /**
     * It cancluate CRC value and return it. coupon parameter must be set code value to calculate CRC value.
     * 
     * @param coupon
     * @return
     */
    static BigInteger calculateCrc(Coupon coupon) {

        BigInteger code = coupon.getCode();
        String crcValue = String.valueOf(code.longValue() % CouponGenerator.CRC_GENERATOR);
        // Debug purpose
        // System.out.println("code.toString(2) : " + code.toString(2));
        // System.out.println("code.longValue() : " + code.longValue());
        // System.out.println("crcValue : " + crcValue);
        return new BigInteger(crcValue, 10);
    }
}




import java.math.BigInteger;
import java.util.Set;

import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;

public class CouponGeneratorTest {

    static CouponGenerator couponGenerator = new CouponGenerator();

    static Set couponSet;

    @BeforeClass
    public static void init() throws CouponGenerateError {

        couponSet = couponGenerator.getCouponSet((byte) 10, 1000);
    }

    /**
     * Test if the CouponGenerator generate requested number of coupons
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCount() throws Exception {

        Assert.assertEquals("Coupon creation didn't match the requested number of coupon", 1000, couponSet.size());
    }

    /**
     * Test if the CodeGenerator generate exact size of vaucher code chracter size.
     * 
     * @throws Exception
     */
    @Test
    public void testCouponCharacterSize() throws Exception {

        for (Coupon coupon : couponSet) {

            // System.out.println(CouponUtil.getVoucherString(coupon));
            Assert.assertEquals("Coupon Characters didn't match the requested number of character", 10, CouponUtil
                    .getVoucherString(coupon).length());
        }

    }

    /**
     * Test if the generated CRC value passes validation check. 
* If it fails, one of generation or validation is wrong * * @throws Exception */ @Test public void testCrcCreation() throws Exception { for (Coupon coupon : couponSet) { Assert.assertTrue("Checksum value is not valid", CouponUtil.isValidCrc(coupon)); } } /** * Test if the given coupon passes CRC validation check. * * @throws Exception */ @Test public void testCrcValidation() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("f", 32)); Assert.assertTrue("CRC value value is not valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the given coupon fails CRC validation check. * * @throws Exception */ @Test public void testCrcValidation2() throws Exception { Coupon coupon = new Coupon(); coupon.setCode(new BigInteger("54321", 32)); // CRC value is 1. ex ) 54321 (32 radix) % 31 = 5377089 (10 radix) % 31 = 15 (F) coupon.setCrc(new BigInteger("2", 32)); Assert.assertFalse("CRC value must not be valid", CouponUtil.isValidCrc(coupon)); } /** * Test if the CouponGenerator accept requesting less than 3 chracters or bigger than 12 chracters. In this case, it * must throw Exception */ @Test public void testCouponMinusCharacterSize() { try { couponGenerator.getCoupon((byte) 2); Assert.fail("small charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) -6); Assert.fail("Minus charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } try { couponGenerator.getCoupon((byte) 13); Assert.fail("Bigger than 12 charater size is not allowed"); } catch (CouponGenerateError e) { Assert.assertTrue(true); } } }


Tuesday, February 16, 2016

Image Resizing by java, Change JPEG image quality

I have seen sample code from others to resize image and change jpeg quality by java. It was pretty good. but, just I didn't like one thing that most of them didn't close the stream of file reading or writing. So, The image files were not able to removed or rewrited. I guess probably, it will cause to create jomby process which is never die unless restart JVM. Thease kind of things must be avoided to make a stable application. It will not be visible for now. but, It will capture the application one day and make difficult to find out why application is not stable.

Add Maven Dependendy to resize image
<dependency>
 <groupId>org.imgscalr</groupId>
 <artifactId>imgscalr-lib</artifactId>
 <version>4.2</version>
</dependency>


import javax.imageio.IIOImage;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.ImageWriteParam;
import javax.imageio.ImageWriter;
import javax.imageio.plugins.jpeg.JPEGImageWriteParam;
import javax.imageio.stream.FileImageOutputStream;

import org.imgscalr.Scalr;
import org.imgscalr.Scalr.Method;
import org.imgscalr.Scalr.Mode;
...

public class ImageUtil {

    public static void downsizeJpg(Path sourcePath, Path targetPath, int targetWidth) throws IOException {

        try (InputStream isSrc = new FileInputStream(sourcePath.toFile())) {

            // Don't Use "ImageIO.read(File file)". It will not close the stream
            // Use Stream to close the resource    
            BufferedImage sourceImage = ImageIO.read(isSrc);

            double width = sourceImage.getWidth();
            double height = sourceImage.getHeight();
            double ratio = width / height;
            int trgHeight = (int) (targetWidth / ratio);

            // If taget width is smaller than source width, start to downsize
            if (targetWidth < width) {

                BufferedImage targetImage = Scalr.resize(sourceImage, Method.QUALITY, Mode.FIT_EXACT, targetWidth,
                        trgHeight);
                saveImage(targetImage, targetPath, 0.7f);

                // prevent upsizing. just copy the source image to target image
            } else {

                saveImage(sourceImage, targetPath, 0.7f);
            }
        }
    }

    private static void saveImage(RenderedImage image, Path targetPath, float quality) throws FileNotFoundException,
            IOException {

        // set compression level 
        JPEGImageWriteParam jpegParams = new JPEGImageWriteParam(null);
        jpegParams.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
        jpegParams.setCompressionQuality(quality);

        final ImageWriter writer = ImageIO.getImageWritersByFormatName("jpg").next();

        try (FileImageOutputStream fios = new FileImageOutputStream(targetPath.toFile())) {

            // specifies where the jpg image has to be written
            writer.setOutput(fios);

            // save the image
            writer.write(null, new IIOImage(image, null, null), jpegParams);
            writer.dispose();
        }

    }
}

quality can be fromo 0f to 1f. 1 is the highst quality and file size get big. 0 is lowest quality and file size get small.

Thursday, May 28, 2015

Print current JVM's Java System Property

It print out all System Properties on console.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.getProperties().list(System.out);
    }
}


Then you pick up one property which you need and use it.
package com.devtrigger;

public class SystemProperty {

    public static void main(String... args) {

        System.out.println(System.getProperty("user.home"));
        System.out.println(System.getProperty("file.separator"));

...

        // create file object which is under user's home directory
        File file = new File (System.getProperty("user.home") + System.getProperty("file.separator") + "setting.xml");
    }
}


Referred from : System Properties

Wednesday, May 20, 2015

a Java Class which scans ports in an IP

I had to run port scanner software for security check reason, before deliver the product. I tried to get one utilillity from internet. but, the Websites were blocked in my company to download. so, I just made a Java Class which can do the port scanning. it takes around 20 min to run throught all the ports. If I increase Thread size and shorten the timeout, it will take less then 5 min.

Fornow, I have a little problem. When I let it run quickly, Firewall or security software get activated and deny to response if the port is opened. So, I had to let it run slowly to find all the opened port.

package com.devtrigger;

import java.net.InetSocketAddress;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;

class PortScanner {

    public static void main(final String... args) throws InterruptedException, ExecutionException {
        final ExecutorService es = Executors.newFixedThreadPool(15);

        final String ip = "127.0.0.1";

        final int timeout = 200;
        final List<Future<ScanResult>> futures = new ArrayList<>();
        for (int port = 1; port <= 65535; port++) {
            futures.add(portIsOpen(es, ip, port, timeout));
        }
        es.awaitTermination(200L, TimeUnit.MILLISECONDS);
        int openPorts = 0;
        for (final Future<ScanResult> f : futures) {
            if (f.get().isOpen()) {
                openPorts++;
                System.out.println(f.get().getPort());
            }
        }
        System.out.println("There are " + openPorts + " open ports on host " + ip + " (probed with a timeout of "
                + timeout + "ms)");
    }

    public static Future<ScanResult> portIsOpen(final ExecutorService es, final String ip, final int port,
            final int timeout) {
        return es.submit(new Callable<ScanResult>() {
            @Override
            public ScanResult call() {
                try {
                    Socket socket = new Socket();
                    socket.connect(new InetSocketAddress(ip, port), timeout);
                    socket.close();
                    return new ScanResult(port, true);
                } catch (Exception ex) {
                    return new ScanResult(port, false);
                }
            }
        });
    }

    public static class ScanResult {
        private int port;

        private boolean isOpen;

        public ScanResult(int port, boolean isOpen) {
            super();
            this.port = port;
            this.isOpen = isOpen;
        }

        public int getPort() {
            return port;
        }

        public void setPort(int port) {
            this.port = port;
        }

        public boolean isOpen() {
            return isOpen;
        }

        public void setOpen(boolean isOpen) {
            this.isOpen = isOpen;
        }

    }
}

Friday, February 6, 2015

How to create and run Apache JMeter Test Scripts from a Java program


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

 <modelVersion>4.0.0</modelVersion>

 <groupId>myportal</groupId>
 <artifactId>loadtest</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>

 <name>Zug Portal Load Test Tool</name>
 <url>http://maven.apache.org</url>

 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.12</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.apache.jmeter</groupId>
   <artifactId>ApacheJMeter_http</artifactId>
   <version>2.11</version>
  </dependency>

 </dependencies>
</project>
package myportal.loadtest;

import java.net.URL;

import org.apache.jmeter.control.LoopController;
import org.apache.jmeter.engine.StandardJMeterEngine;
import org.apache.jmeter.protocol.http.sampler.HTTPSampler;
import org.apache.jmeter.testelement.TestElement;
import org.apache.jmeter.testelement.TestPlan;
import org.apache.jmeter.threads.SetupThreadGroup;
import org.apache.jmeter.util.JMeterUtils;
import org.apache.jorphan.collections.HashTree;

public class SampleJmeter {

    public static void main(String[] args) {
        // Engine
        StandardJMeterEngine jm = new StandardJMeterEngine();
        URL property = SampleJmeter.class.getClassLoader().getResource("jmeter.properties");

        // jmeter.properties
        JMeterUtils.loadJMeterProperties(property.getPath());

        HashTree hashTree = new HashTree();

        // HTTP Sampler
        HTTPSampler httpSampler = new HTTPSampler();
        httpSampler.setDomain("www.google.com");
        httpSampler.setPort(80);
        httpSampler.setPath("/");
        httpSampler.setMethod("GET");

        // Loop Controller
        TestElement loopCtrl = new LoopController();
        ((LoopController) loopCtrl).setLoops(1);
        ((LoopController) loopCtrl).addTestElement(httpSampler);
        ((LoopController) loopCtrl).setFirst(true);

        // Thread Group
        SetupThreadGroup threadGroup = new SetupThreadGroup();
        threadGroup.setNumThreads(1);
        threadGroup.setRampUp(1);
        threadGroup.setSamplerController((LoopController) loopCtrl);

        // Test plan
        TestPlan testPlan = new TestPlan("MY TEST PLAN");

        hashTree.add("testPlan", testPlan);
        hashTree.add("loopCtrl", loopCtrl);
        hashTree.add("threadGroup", threadGroup);
        hashTree.add("httpSampler", httpSampler);

        jm.configure(hashTree);

        jm.run();
    }
}

meter.properties file is from the JMeter installation /bin directory.

Monday, May 5, 2014

Android infrared (IR) transmitter code sample for Kitkat and Jelly Bean

There are many remote controller application in Google play store. But, I was not able to find an app that I can put my own IR pattern. For some reason, I need to transmit my own pattern.

So, I made a remote controller application. Now I can reach my purpose. It was easy to make for Kitkat OS. but, I wanted to make it run for Jelly Bean OS. Despite of Android API officially offer IR API from Kitkat(4.4). apparently, I see few Samsung Android device has IR transmitter which is running Jelly Bean OS(4.2).

When I run the same app which is using IR API. It didn't run on Jelly Bean as expected. I was wonder how I can control the IR from Jelly Bean OS. Luckily, I found a code sample from https://github.com/rngtng/IrDude thanks to him.

There are list of manufactur IR pattern. Please refer http://www.remotecentral.com/cgi-bin/codes/


\AndroidManifest.xml
Just make sure that minSdkVersion is 17 and target version is 19. as well it needs a permission and feature to handle IR transmitter.
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.devtrigger.remotecontrol"
    android:versionCode="1"
    android:versionName="1.0" >

    <uses-permission android:name="android.permission.TRANSMIT_IR"
    android:required="false" />
    <uses-feature android:name="android.hardware.consumerir" />
    <uses-sdk
        android:minSdkVersion="17"
        android:targetSdkVersion="19" />

    <application
        android:allowBackup="true"
        android:icon="@drawable/ic_launcher"
        android:label="@string/app_name"
        android:theme="@style/AppTheme" >
        <activity
            android:name="com.devtrigger.remotecontrol.MainActivity"
      android:screenOrientation="portrait"
            android:label="@string/app_name" >
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

</manifest>



\res\layout\activity_main.xml
There are just power , Channel up and down button.
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical" >

    <Button
        android:id="@+id/buttonPower"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:onClick="irSend"
        android:text="Power" />

    <Button
        android:id="@+id/buttonChUp"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:onClick="irSend"
        android:text="CH +" />

    <Button
        android:id="@+id/buttonChDown"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:onClick="irSend"
        android:text="CH -" />

</LinearLayout>


\src\com\devtrigger\remotecontrol\MainActivity.java
I just removed my own pattern to generalize it and share it on my blog. I put Samsung TV IR pattern on below sample and this Class works find for my Samsung Android device. I have test Galaxy S4 mini Jelly Bean and Galaxy S4 Kitkat.
package com.devtrigger.remotecontrol;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import android.annotation.TargetApi;
import android.app.Activity;
import android.content.Context;
import android.hardware.ConsumerIrManager;
import android.os.Build;
import android.os.Bundle;
import android.util.SparseArray;
import android.view.View;
import android.widget.TextView;

public class MainActivity extends Activity {

    Object irdaService;
    Method irWrite;    
    SparseArray<String> irData;
    TextView mFreqsText;
    ConsumerIrManager mCIR;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        // Be sure to call the super class.
        super.onCreate(savedInstanceState);

        setContentView(R.layout.activity_main);

        irData = new SparseArray<String>();
  irData.put(
    R.id.buttonPower,
    hex2dec("0000 006d 0022 0003 00a9 00a8 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0040 0015 0015 0015 003f 0015 003f 0015 003f 0015 003f 0015 003f 0015 003f 0015 0702 00a9 00a8 0015 0015 0015 0e6e"));
  irData.put(
    R.id.buttonChUp,
    hex2dec("0000 006d 0022 0003 00a9 00a8 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 0015 0015 0015 0015 003f 0015 0015 0015 0015 0015 0015 0015 003f 0015 0015 0015 003f 0015 003f 0015 0015 0015 0040 0015 003f 0015 003f 0015 0702 00a9 00a8 0015 0015 0015 0e6e"));
  irData.put(
    R.id.buttonChDown,
    hex2dec("0000 006d 0022 0003 00a9 00a8 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 003f 0015 003f 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 0015 003f 0015 0015 0015 0015 0015 0015 0015 003f 0015 003f 0015 003f 0015 003f 0015 0015 0015 003f 0015 003f 0015 003f 0015 0702 00a9 00a8 0015 0015 0015 0e6e"));

  
   if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT){
    
    irInit4KitKat();
   }else{
    irInit4JellyBean();
   }
  
    }
    
    @TargetApi(Build.VERSION_CODES.KITKAT)
    public void irInit4KitKat() {
     
     // Get a reference to the ConsumerIrManager
        mCIR = (ConsumerIrManager)getSystemService(Context.CONSUMER_IR_SERVICE);
 
 }

 public void irInit4JellyBean() {
  irdaService = this.getSystemService("irda");
  Class c = irdaService.getClass();
  Class p[] = { String.class };
  try {
   irWrite = c.getMethod("write_irsend", p);
  } catch (NoSuchMethodException e) {
   e.printStackTrace();
  }
 }

 public void irSend(View view) {
  
  if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT){
   
   irSend4Kitkat(view);
  }else{
   
   irSend4JellyBean(view);
  }
 }
 
 @TargetApi(Build.VERSION_CODES.KITKAT)
 private void irSend4Kitkat(View view) {

    
  String data = irData.get(view.getId());
  if (data != null) {
   String values[] = data.split(",");
   int[] pattern = new int[values.length-1];
   
   for (int i=0; i<pattern.length; i++){
    pattern[i] = Integer.parseInt(values[i+1]);
   }
   
   mCIR.transmit(Integer.parseInt(values[0]), pattern);
  }
 }
 
 private void irSend4JellyBean(View view) {
  String data = irData.get(view.getId());
  if (data != null) {
   try {
    irWrite.invoke(irdaService, data);
   } catch (IllegalArgumentException e) {
    e.printStackTrace();
   } catch (IllegalAccessException e) {
    e.printStackTrace();
   } catch (InvocationTargetException e) {
    e.printStackTrace();
   }
  }
 }

 protected String hex2dec(String irData) {
  List<String> list = new ArrayList<String>(Arrays.asList(irData
    .split(" ")));
  list.remove(0); // dummy
  int frequency = Integer.parseInt(list.remove(0), 16); // frequency
  list.remove(0); // seq1
  list.remove(0); // seq2

  for (int i = 0; i < list.size(); i++) {
   list.set(i, Integer.toString(Integer.parseInt(list.get(i), 16)));
  }

  frequency = (int) (1000000 / (frequency * 0.241246));
  list.add(0, Integer.toString(frequency));

  irData = "";
  for (String s : list) {
   irData += s + ",";
  }
  return irData;
 }
}

Friday, March 14, 2014

Reversed Binary Numbers (Difficulty Level: Easy)

My friend sent me a quiz how to make a reverse binary from input number and put it back as number.
It is easy level though, I had fun of it
If you are interested to get more quiz, please check this https://code.google.com/codejam/
There are many programing quiz in Internet. I see there are so many genius in the world.
I wonder when I can resolve the difficult level of quiz. I must study math again.


Task

Your task will be to write a program for reversing numbers in binary. For instance, the binary representation of 13 is 1101, and reversing it gives 1011, which corresponds to number 11.

Input

The input contains a single line with an integer N, 1 ≤ N ≤ 1000000000.

Output

Output one line with one integer, the number we get by reversing the binary representation of N.

Sample input 1
13
Sample output 1
11
Sample input 2
47
Sample output 2
61




package puzzle;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

/**
 *
 * @author jack
 */
public class Reversebinary {

    /**
     * @param args the command line arguments
     */
    public static void main(String args[]) {

        try {
            BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

            String input;

            while ((input = br.readLine()) != null) {
                int inputValue = Integer.parseInt(input);

//                System.out.println(Integer.toBinaryString(inputValue));

                StringBuffer revertedString = new StringBuffer();
                revertedString.append(Integer.toBinaryString(inputValue)).reverse();

//                System.out.println(revertedString);
                System.out.println(Long.parseLong(revertedString.toString(), 2));

            }

        } catch (java.lang.NumberFormatException ne) {
            System.out.println("Please type numbers");
        } catch (IOException io) {
            io.printStackTrace();

        }
    }
}

How to get the apk out of the non-rooted Android device

I got a request to test updated Android application. but, it often happens that I need to test without apk file.
Due to complicated commpany rules, politics and security, the requester can't send me the apk file.. But, they push me to do..
To resolve this situation, luckily I had one Android device which is installed the latest apk. but, I need to test it on another device.
I can extract the apk from an Anroid device and install it on the other devices.

To do this step, Android SDK is required. so that you will be able use 'adb' command.

Step to extract apk file from an Android device.
1. Connect the device to the computer
2. start command prompt or shell
3. adb shell pm list package -f -3 (to display all the installed apps on the device)
C:\>adb shell pm list package -f -3
package:/data/app/autotechniksteeg.preistafel-1.apk=autotechniksteeg.preistafel
...
package:/data/app/uk.amazon.mShop.android-2.apk=uk.amazon.mShop.android
...

4. adb pull {apk name} (This will save the required apk on the current directory of the computer)
C:\>adb pull /data/app/uk.amazon.mShop.android-2.apk


If you remove '-3' option, it will show all apk which is pre-loaded app on your device. I used 'adb shell pm list package -f -3' command.
If you like to know more about 'pm list package' filter option, please refer below.
pm list packages: prints all packages, optionally only
  those whose package name contains the text in FILTER.  Options:
    -f: see their associated file.
    -d: filter to only show disbled packages.
    -e: filter to only show enabled packages.
    -s: filter to only show system packages.
    -3: filter to only show third party packages.
    -i: see the installer for the packages.
    -u: also include uninstalled packages.

Thursday, October 24, 2013

receive argument on expect shell script

I just wanted to scp easily. I just simply made a expect shell script which receive one argument.
I just excute one script like below

cp.sh file

and copy finished
#!/usr/bin/expect -f

set filename [lindex $argv 0];

spawn scp -v /temp/$filename root@192.168.1.1:/tmp/root/
expect "Enter passphrase"
send "\r"
...

Thursday, August 8, 2013

Manipulate Access Control List (ACL) on mysql

I had to limit of access to mysql server due to security improvement task.
We like to allow only few host to the mysql. It was very easy

mysql>use mysql
mysql> select host, user from user;
----------------------------+
host  user
----------------------------+
%  dmitry
host : % means all (It is security breach)
host side should be ip address of web server
or
ip address of admin pc.
example)
-- Replace unlimited access rule to only allow 192.168.0.5 host rule
mysql>update user set host='192.168.0.5' where host = '%' and user in ('dmitry');
mysql>commit;
mysql>flush privileges;
Query OK, 0 rows affected (0.01 sec)


-- Remove unlimited access rule

mysql> delete from user where host = '%';
mysql> commit;
mysql> flush privileges;


you can insert more hosts if you like to.

Monday, July 29, 2013

Android Json file read instead of property file read


While I am building an Android application. I wanted to read a property file and wanted to act my app depends on the property file configuration. But, property file can't handle array unless I add dependency Apache Common Configuration. As I am building an mobile application, it should be compact. for the reason, I just used Android JSONObject to resolve this.

Create "country.json" file and locate it Android device "/" directory

{
    "country": [
        {
            "name": "germany",
            "code": "DE",
            "continent": "Europe",
            "eat": "sausage"
        },    
        {
            "name": "france",
            "code": "FR",
            "continent": "Europe",
            "eat": "croissant"
        },
        {
            "name": "korea",
            "code": "KR",
            "continent": "ASIA",
            "eat": "rice"
        },
        {
            "name": "japan",
            "code": "JP",
            "continent": "ASIA",
            "eat": "fish"
        }
    ]
}


Create 2 method and use "getCountryList()" on your need.
private List<Country> getCountryList() throws Exception {

 File dirSDCard = Environment.getExternalStorageDirectory();
 File yourFile = new File(dirSDCard, "country.json");
 InputStream jsonStream = new FileInputStream(yourFile);
 JSONObject jsonObject = new JSONObject(InputStreamToString(jsonStream));
 JSONArray jsonArray = jsonObject.getJSONArray("country");
 List<Country> countryList = new ArrayList<Country>();

 for (int i = 0; i < jsonArray.length(); i++) {

  JSONObject jsonCountry = jsonArray.getJSONObject(i);
  Country country = new Country();
  country.setName(jsoncountry.getString("name"));
  country.setCode(jsoncountry.getString("code"));
  country.setContinent(jsoncountry.getString("continent"));
  country.setEat(jsoncountry.getString("eat"));

  countryList.add(country);
 }

 return countryList;
}

private String InputStreamToString(InputStream is) {

 BufferedReader r = new BufferedReader(new InputStreamReader(is));
 StringBuilder total = new StringBuilder();
 String line;
 try {
  while ((line = r.readLine()) != null) {
   total.append(line);
  }
 } catch (IOException e) {
  e.printStackTrace();
 }
 return total.toString();

}

Monday, June 17, 2013

mybatis various datasource sample jndi, hsql, oracle, mysql, sqlite configuration setup with Springframework

While doing various projects, I had to use different DBs. I just summarized datasource for Mybatis samples which is after testing. You can simply comment out to for your prefer datasource.
If you like to know detail to implement this sample with Spring framework, please follow mybatis-integration-with-spring.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
 xmlns:context="http://www.springframework.org/schema/context"
 xmlns:jdbc="http://www.springframework.org/schema/jdbc"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context-3.0.xsd
    http://www.springframework.org/schema/jdbc
        http://www.springframework.org/schema/jdbc/spring-jdbc-3.0.xsd">

<!-- JNDI datasource -->
<!-- 
<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
 <property name="jndiName" value="jdbc/oraclePool" />
 <property name="resourceRef" value="true" />
</bean>
-->

<!-- Hsql datasource -->
<!-- 
<jdbc:embedded-database id="dataSource">
 <jdbc:script location="classpath:hsql/schema.sql" />
 <jdbc:script location="classpath:hsql/data.sql" />
</jdbc:embedded-database>
-->

<!-- Oracle JDBC datasource -->
<!-- 
<bean id="dataSource"
 class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
 <property name="driverClass" value="oracle.jdbc.OracleDriver" />
 <property name="url" value="jdbc:oracle:thin:@HOST:1111:ORA" />
 <property name="username" value="ID" />
 <property name="password" value="PASSWORD" />
</bean>
-->

<!-- mysql JDBC datasource -->
<!-- 
<bean id="dataSource"
 class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
 <property name="driverClass" value="com.mysql.jdbc.Driver" />
 <property name="url" value="jdbc:mysql://localhost:3306/mydata" />
 <property name="username" value="user123" />
 <property name="password" value="12345678" />
</bean>
 -->

<!-- sqlite JDBC datasource -->
<!-- 
<bean id="dataSource"
 class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
 <property name="driverClass" value="org.sqlite.JDBC" />
 <property name="url" value="jdbc:sqlite:C:/yourdb.sqlite" />
</bean>
 -->

<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
 <property name="dataSource" ref="dataSource" />
 <property name="typeAliasesPackage" value="com.devtrigger.model" />
 <property name="mapperLocations" value="classpath*:dao/**/*.xml" />
</bean>

<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
 <constructor-arg index="0" ref="sqlSessionFactory" />
</bean>

<!-- scan for mapper interface files and let them be autowired -->
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
 <property name="basePackage" value="com.devtrigger.dao" />
</bean>

<bean id="transactionManager"
 class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
 <property name="dataSource" ref="dataSource" />
</bean>

</beans>

Thursday, March 14, 2013

Useful SVN adminitrator command

Export svn
- When I don't know SVN credential. but I have access to repository server
$svn export --force file:///home/svn/myrepos ./myrepos


Check User access control
$cat /home/svn/myrepos/conf/authz
$cat /home/svn/myrepos/conf/passwd
$cat /home/svn/myrepos/conf/svnserve.conf


Create version 1.6 compatible Repository
$svnadmin create myrepos --pre-1.6-compatible


Dump svn repository
$svnadmin dump /home/svn/myrepos > /home/backup/svn/myrepos.dump


Incremental dump svn repository
- SVN incremental Backup
$ svnadmin dump myrepos --revision 0:1000 > dumpfile1
$ svnadmin dump myrepos --revision 1001:2000 --incremental > dumpfile2
$ svnadmin dump myrepos --revision 2001:3000 --incremental > dumpfile3
$ svnadmin dump <repos> -r 58:HEAD --deltify > <file2> 


Import dump file
$ cd /home/svn
$ svnadmin load --bypass-prop-validation myrepos < /home/backup/svn/myrepos.dump


Incremental import dump file
- SVN recovery 
* incremental :
$ svnadmin load < ~/repos-0-1000.svn_dump
$ svnadmin load < ~/repos-1000-2000.svn_dump
$ svnadmin load < ~/repos-2000-3000.svn_dump


Kill SVN daemon on Solaris
$ pkill -KILL svnserve


Start SVN daemon
$ svnserve -d -r /home/svn/


Pack svn
- SVN revision pack per 1000 revision
$ svnadmin pack /home/svn/sfc


Tuesday, December 11, 2012

Mybatis integration with Spring 3.1 Framework.

Hi All,

I have configured new web project with mybatis and springframework several times. due to several project and being asked from team member to set up another project springframework skeleton. There are many advantage to start up from scratch. then you will know what it all means. But, When you need a quick start up, it would be good to have a project template.

As I have configured it several times. I improved of understanding of these framework setting then I made simple optimized web project template for share with others.

This sample will show the result of Json String.

I recommend you to look through below source roughly and download attached sample source to test running and understanding. it will run fine on tomcat I have tested. you will not need any db connection for sample code. below sample is running based on memory DB Hsql.

Sample Source is attached.
you can download from below link
Download

To integrate mybatis with Spring framework I recommend you to follow below step :

1. create java web project.
2. get all dependencies and configure classpath ( I am going to use maven )
2. configure springframework.
3. configure mybatis
4. create classes and xml ( Mapper interface, Mapper xml, Spring controller, create beans)
5. run it on tomcat or any WAS
6. check result


Directory Structure, you can refer below



Dependency Overview :




Maven Dependency : pom.xml
Springframework 3.1.3, Jackson Mapper 1.9.11, Hsql 2.2.8
I have excluded mybatis spring dependency which depends on old verion of springframework

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.devtrigger</groupId>
 <artifactId>web-template</artifactId>
 <version>1.0-SNAPSHOT</version>
 <packaging>war</packaging>

 <name>Web Template Project, you can copy this and use for new project set up quickly</name>

 <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <spring-version>3.1.3.RELEASE</spring-version>
 </properties>

 <dependencies>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-orm</artifactId>
   <version>${spring-version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-webmvc-portlet</artifactId>
   <version>${spring-version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-oxm</artifactId>
   <version>${spring-version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-test</artifactId>
   <version>${spring-version}</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.mybatis</groupId>
   <artifactId>mybatis-spring</artifactId>
   <version>1.1.1</version>
   <exclusions>
    <exclusion>
     <groupId>org.springframework</groupId>
     <artifactId>spring-context</artifactId>
    </exclusion>
    <exclusion>
     <groupId>org.springframework</groupId>
     <artifactId>spring-core</artifactId>
    </exclusion>
    <exclusion>
     <groupId>org.springframework</groupId>
     <artifactId>spring-jdbc</artifactId>
    </exclusion>
    <exclusion>
     <groupId>org.springframework</groupId>
     <artifactId>spring-test</artifactId>
    </exclusion>
    <exclusion>
     <groupId>org.springframework</groupId>
     <artifactId>spring-tx</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>log4j</groupId>
   <artifactId>log4j</artifactId>
   <version>1.2.17</version>
  </dependency>
  <dependency>
   <groupId>org.codehaus.jackson</groupId>
   <artifactId>jackson-mapper-asl</artifactId>
   <version>1.9.11</version>
  </dependency>
  <dependency>
   <groupId>javax.servlet</groupId>
   <artifactId>jstl</artifactId>
   <version>1.2</version>
  </dependency>
  <dependency>
   <groupId>javax.servlet</groupId>
   <artifactId>servlet-api</artifactId>
   <version>2.5</version>
   <scope>provided</scope>
  </dependency>
  <dependency>
   <groupId>javax</groupId>
   <artifactId>javaee-web-api</artifactId>
   <version>6.0</version>
   <scope>provided</scope>
  </dependency>
  <dependency>
   <groupId>org.hsqldb</groupId>
   <artifactId>hsqldb</artifactId>
   <version>2.2.8</version>
  </dependency>
 </dependencies>

 <build>
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
     <source>1.6</source>
     <target>1.6</target>
     <encoding>utf-8</encoding>
    </configuration>
   </plugin>
   <plugin>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.2</version>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <version>2.8</version>
    <configuration>
     <additionalProjectFacets>
      <jst.web>2.5</jst.web>
     </additionalProjectFacets>
     <downloadJavadocs>true</downloadJavadocs>
     <downloadSources>true</downloadSources>
     <wtpContextName>/</wtpContextName>
     <wtpdefaultserver>${eclipse.wtpdefaultserver}</wtpdefaultserver>
     <wtpversion>2.0</wtpversion>
    </configuration>
   </plugin>
  </plugins>
 </build>
</project>


Mybatis configuration : mybatis-context.xml
If you comment out or in, you can change datasource.
I have prepared sample for JDNI, JDBC and Hsql datasources which are all tested. Currently, It is running on Hsql for simple sample running on your memory based DB.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
 xmlns:context="http://www.springframework.org/schema/context"
 xmlns:jdbc="http://www.springframework.org/schema/jdbc"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context-3.0.xsd
  http://www.springframework.org/schema/jdbc
        http://www.springframework.org/schema/jdbc/spring-jdbc-3.0.xsd">

 <!-- JNDI datasource -->
 <!-- 
 <bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean"> 
  <property name="jndiName" value="jdbc/oraclePool"/>
  <property name="resourceRef" value="true" />
 </bean> 
  -->
 
 <!-- Hsql datasource -->
 <!--
 -->
 <jdbc:embedded-database id="dataSource" >
  <jdbc:script location="classpath:hsql/schema.sql"/>
  <jdbc:script location="classpath:hsql/data.sql"/>
 </jdbc:embedded-database>

 <!-- JDBC datasource -->
 <!--
 <bean id="dataSource"
  class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
  <property name="driverClass" value="oracle.jdbc.OracleDriver" />
  <property name="url" value="jdbc:oracle:thin:@HOST:1111:ORA" />
  <property name="username" value="ID" />
  <property name="password" value="PASSWORD" />
 </bean>
 -->

 <bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
  <property name="dataSource" ref="dataSource" />
  <property name="typeAliasesPackage" value="com.devtrigger.model" />
  <property name="mapperLocations" value="classpath*:dao/**/*.xml" />
 </bean>

 <bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
  <constructor-arg index="0" ref="sqlSessionFactory" />
 </bean>

 <!-- scan for mapper interface files and let them be autowired -->
 <bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
  <property name="basePackage" value="com.devtrigger.dao" />
 </bean>

 <bean id="transactionManager"
  class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
  <property name="dataSource" ref="dataSource" />
 </bean>

</beans>



XML Mapper : JobMapper.xml
You need to write your actual sql queries. value will be replace by "#{property}"
"<property name="typeAliasesPackage" value="com.devtrigger.model" />" is configured on mybatis-context.xml to make it short to write of bean name. You would need to write "com.devtrigger.model.JobInfo" for using a java bean or need to be used typeAlias. Please aware that "jobInfo" is id of result map, "JobInfo" is a java bean.
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper
  PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
  "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.devtrigger.dao.JobMapper">

 <cache />

 <resultMap type="JobInfo" id="jobInfo">
  <result property="id" column="id" />
  <result property="panNumber" column="pan_number" />
  <result property="address" column="address" />
  <result property="city" column="city" />
  <result property="state" column="state" />
  <result property="pincode" column="pincode" />
 </resultMap>

 <select id="selectAll" resultMap="jobInfo">
  <![CDATA[
  select id, pan_number, address, city, state, pincode
  from JBT_MEM
  ]]>
 </select>

 <insert id="insert" parameterType="JobInfo">
  <![CDATA[
  insert into JBT_MEM values(#{id}, #{pan_number}, #{address}, #{city}, #{state}, #{pincode})
  ]]>  
 </insert>

 <update id="update" parameterType="JobInfo">
  <![CDATA[
  ]]> 
 </update>


 <delete id="delete" parameterType="JobInfo">
  <![CDATA[
  ]]> 
 </delete>
</mapper>


Mapper Interface : JobInfo.java
you need to have interface for mapping with Mapper xml file. the method name id must mapped with ID of SQL described in the mapper xml.
package com.devtrigger.dao;

import java.util.List;

import com.devtrigger.model.JobInfo;

public interface JobMapper {

 List<JobInfo> selectAll();
 int insert(JobInfo sampleInfo);
 int update(JobInfo sampleInfo);
 int delete(JobInfo sampleInfo);
}


Spring Controller : IndexController.java

package com.devtrigger.controller;

import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

import com.devtrigger.model.JobInfo;
import com.devtrigger.service.JobService;

@Controller
public class IndexController {

 private final Log log = LogFactory.getLog(this.getClass());

 @Autowired
 private JobService jobService;
 
 @RequestMapping(value = "/index")
 @ResponseBody
 public List<JobInfo> requestIndex(){
  
  log.debug("controller is called");
  return jobService.getSampleInfo();
 }
 
}



type URL "http://localhost:8080/index.do" on your browser then you will see Json String returns. The reason I show Json result instead of jsp servlet result is, because you will find many sample on internet for jsp servlet result return
BTW, you need to put your jsp file under "/WEB-INF/jsp/" if you need.

[{"id":"1","panNumber":"AABBAABB","address":"Address","city":"NY","state":"AB","pincode":23500},{"id":"2","panNumber":"BGDBCBDB","address":"Address","city":"LA","state":"St","pincode":23500}]