8/3/17

Setting up Tomcat Cluster for SpagoBI 5.1.

Setting up SpagoBI 5.1 on Tomcat with MySQL is in the previous post.
Details on Tomcat Clustering.

Objective

Create a Cluster of Tomcat running SpagoBI, so that we can have a redundant setup. We will be setting up session based clustering as this is quick and easy.

Setup Overview as below.

                             /--> spagobi-node-1
                            /--> spagobi-node-2
Apache (httpd)  -----[RoundRobin]----> spagobi-node-3
                            \--> spagobi-node-4
                             \--> spagobi-node-5

Configuration port information for each spagobi Instance.

We will be configuring all the instance on the same server, for this test.
NOTE : In a real environment we will have multiple servers, each hosting a single instance of SpagoBI. In that case we will not need to do the below port distribution.
                SHUTDOWN    AJP        SSL
spagobi-node-1    8005        8009    8443
spagobi-node-2    8015        8019    8444
spagobi-node-3    8025        8029    8445
spagobi-node-4    8035        8039    8446
spagobi-node-5    8045        8049    8447

Create Multiple Instance of the existing tomcat.

We will be creating multiple instance of the same server in /opt
[ahmed@ahmed-server ~]# cd /opt/
[ahmed@ahmed-server opt]# cp -rf apache-tomcat-7.0.59 apache-tomcat-7.0.59-8019
[ahmed@ahmed-server opt]# cp -rf apache-tomcat-7.0.59 apache-tomcat-7.0.59-8029
[ahmed@ahmed-server opt]# cp -rf apache-tomcat-7.0.59 apache-tomcat-7.0.59-8039
[ahmed@ahmed-server opt]# cp -rf apache-tomcat-7.0.59 apache-tomcat-7.0.59-8049
[ahmed@ahmed-server opt]# mv apache-tomcat-7.0.59 apache-tomcat-7.0.59-8009
Here is how the new directories look like.

Updating server.xml file.

We will first update the base 8009 Instance and later copy to all other instances.
vim /opt/apache-tomcat-7.0.59-8009/conf/server.xml

Setting up the SHUTDOWN port.

We will be setting the below port for each server for SHUTDOWN
spagobi-node-1    8005
spagobi-node-2    8015
spagobi-node-3    8025
spagobi-node-4    8035
spagobi-node-5    8045
Make the change in the tag as below.

Here is how it looks on the server.xml
"Shutdown Port"

Comment below tag as we will not be using this.


Here is how it looks for my setup.
"Disable Port 8080"

Update spago_service_url and spago_host_url.

Change the server and host url to point to the server on port 80.
As all the services will be served from this port.


Change to ipaddress of the Server or locahost.


Her is how it looks like.
"Change Spago"

Setting AJP port.

Below are the ports which we will be setting for each server.
                AJP        SSL
spagobi-node-1    8009    8443
spagobi-node-2    8019    8444
spagobi-node-3    8029    8445
spagobi-node-4    8039    8446
spagobi-node-5    8049    8447
Here is how it would look for spagobi-node-1, will need to do it for other instances as well.

Setting name for our node.

As stated above AJP 8009, is spagobi-node-1 so we add jvmRoute="spagobi-node-1" to below tag.

Here is how the server.xml looks like.
"AJP and Node Name"

Making Changes to rest of the server.xml.

First lets copy server.xml to all the other instance.
cp /opt/apache-tomcat-7.0.59-8009/conf/server.xml /opt/apache-tomcat-7.0.59-80[1-4]9/conf/
cp /opt/apache-tomcat-7.0.59-8009/conf/server.xml /opt/apache-tomcat-7.0.59-80[123]9/conf/
cp /opt/apache-tomcat-7.0.59-8009/conf/server.xml /opt/apache-tomcat-7.0.59-8019/conf/
cp /opt/apache-tomcat-7.0.59-8009/conf/server.xml /opt/apache-tomcat-7.0.59-8029/conf/
Update SHUTDOWN port on all the instance.
sed -i -- 's/8005/8015/g' /opt/apache-tomcat-7.0.59-8019/conf/server.xml
sed -i -- 's/8005/8025/g' /opt/apache-tomcat-7.0.59-8029/conf/server.xml
sed -i -- 's/8005/8035/g' /opt/apache-tomcat-7.0.59-8039/conf/server.xml
sed -i -- 's/8005/8045/g' /opt/apache-tomcat-7.0.59-8049/conf/server.xml
Update AJP port.
sed -i -- 's/8009/8049/g' /opt/apache-tomcat-7.0.59-8049/conf/server.xml
sed -i -- 's/8009/8039/g' /opt/apache-tomcat-7.0.59-8039/conf/server.xml
sed -i -- 's/8009/8029/g' /opt/apache-tomcat-7.0.59-8029/conf/server.xml
sed -i -- 's/8009/8019/g' /opt/apache-tomcat-7.0.59-8019/conf/server.xml
Update SSL port.
sed -i -- 's/8443/8444/g' /opt/apache-tomcat-7.0.59-8019/conf/server.xml
sed -i -- 's/8443/8445/g' /opt/apache-tomcat-7.0.59-8029/conf/server.xml
sed -i -- 's/8443/8446/g' /opt/apache-tomcat-7.0.59-8039/conf/server.xml
sed -i -- 's/8443/8447/g' /opt/apache-tomcat-7.0.59-8049/conf/server.xml
Update node-name for all the Instance. This is important as this will identify each instance.
sed -i -- 's/node-1/node-2/g' /opt/apache-tomcat-7.0.59-8019/conf/server.xml
sed -i -- 's/node-1/node-3/g' /opt/apache-tomcat-7.0.59-8029/conf/server.xml
sed -i -- 's/node-1/node-4/g' /opt/apache-tomcat-7.0.59-8039/conf/server.xml
sed -i -- 's/node-1/node-5/g' /opt/apache-tomcat-7.0.59-8049/conf/server.xml

Configuring httpd Server.

Assuming httpd is already installed. If not the execute sudo yum install httpdon centos.
Add the below configuration to httpd.conf file in the end.
Location : /etc/https/conf/https.conf

  BalancerMember ajp://127.0.0.1:8009 min=10 max=100 route=spagobi-node-1 loadfactor=1
  BalancerMember ajp://127.0.0.1:8019 min=10 max=100 route=spagobi-node-2 loadfactor=1
  BalancerMember ajp://127.0.0.1:8029 min=10 max=100 route=spagobi-node-3 loadfactor=1
  BalancerMember ajp://127.0.0.1:8039 min=10 max=100 route=spagobi-node-4 loadfactor=1
  BalancerMember ajp://127.0.0.1:8049 min=10 max=100 route=spagobi-node-5 loadfactor=1


ProxyPass /* balancer://spagobicluster/*
[OPTIONAL] we can give the below configuration.
ProxyPass /SpagoBI balancer://spagobicluster/SpagoBI
ProxyPass /SpagoBIAccessibilityEngine balancer://spagobicluster/SpagoBIAccessibilityEngine
ProxyPass /SpagoBIBirtReportEngine balancer://spagobicluster/SpagoBIBirtReportEngine
ProxyPass /SpagoBIChartEngine balancer://spagobicluster/SpagoBIChartEngine
ProxyPass /SpagoBICockpitEngine balancer://spagobicluster/SpagoBICockpitEngine
ProxyPass /SpagoBICommonJEngine balancer://spagobicluster/SpagoBICommonJEngine
ProxyPass /SpagoBIConsoleEngine balancer://spagobicluster/SpagoBIConsoleEngine
ProxyPass /SpagoBIDataMiningEngine balancer://spagobicluster/SpagoBIDataMiningEngine
ProxyPass /SpagoBIGeoEngine balancer://spagobicluster/SpagoBIGeoEngine
ProxyPass /SpagoBIGeoReportEngine balancer://spagobicluster/SpagoBIGeoReportEngine
ProxyPass /SpagoBIJasperReportEngine balancer://spagobicluster/SpagoBIJasperReportEngine
ProxyPass /SpagoBIJPivotEngine balancer://spagobicluster/SpagoBIJPivotEngine
ProxyPass /SpagoBIMobileEngine balancer://spagobicluster/SpagoBIMobileEngine
ProxyPass /SpagoBINetworkEngine balancer://spagobicluster/SpagoBINetworkEngine
ProxyPass /SpagoBIQbeEngine balancer://spagobicluster/SpagoBIQbeEngine
ProxyPass /SpagoBISocialAnalysis balancer://spagobicluster/SpagoBISocialAnalysis
ProxyPass /SpagoBITalendEngine balancer://spagobicluster/SpagoBITalendEngine
ProxyPass /SpagoBIWhatIfEngine balancer://spagobicluster/SpagoBIWhatIfEngine
To add a balancer-manager so that you can see the nodes add the below tag in the end of https.conf file.

  SetHandler balancer-manager
  AuthType Basic
  AuthName "Balancer Manager"
  AuthUserFile "/etc/httpd/conf/.htpasswd"
  Require valid-user

Setting passwd for balancer-manager is simple. Execute below command to create a password for balancer-manager.
[ahmed@ahmed-server ~]# htpasswd -c /etc/httpd/conf/.htpasswd admin
Here is how httpd.conf looks like.

Starting httpd Deamon.

Restarting https using below command.
[ahmed@ahmed-server ~]# service httpd restart

Starting tomcat Instances.

We are start all the instance at once using the below script.
for item in 8009 8019 8029 8039 8049;
do
    /opt/apache-tomcat-7.0.59-${item}/bin/start.sh
    sleep 100
done;
Once we have all the instance started, we can check the balancer-manager on the url as http:/localhost/balancer-manager. Here is how it looks on current setup.
"Balancer Manager"

5/26/17

PRESTO https://prestodb.io/

WHAT IS PRESTO?

Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.
Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook.

WHAT CAN IT DO?

Presto allows querying data where it lives, including Hive, Cassandra, relational databases or even proprietary data stores. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization.
Presto is targeted at analysts who expect response times ranging from sub-second to minutes. Presto breaks the false choice between having fast analytics using an expensive commercial solution or using a slow "free" solution that requires excessive hardware.

WHO USES IT?

Facebook uses Presto for interactive queries against several internal data stores, including their 300PB data warehouse. Over 1,000 Facebook employees use Presto daily to run more than 30,000 queries that in total scan over a petabyte each per day.
Leading internet companies including Airbnb and Dropbox are using Presto.
Presto is amazing. Lead engineer Andy Kramolisch got it into production in just a few days. It's an order of magnitude faster than Hive in most our use cases. It reads directly from HDFS, so unlike Redshift, there isn't a lot of ETL before you can use it. It just works.
Christopher Gutierrez, Manager of Online Analytics, Airbnb
We're really excited about Presto. We're planning on using it to quickly gain insight about the different ways our users use Dropbox, as well as diagnosing problems they encounter along the way. In our tests so far it's been rock solid and extremely fast when applied to some of our most important ad hoc use cases.
Fred Wulff, Software Engineer, Dropbox

5/8/17

API developer cookbook

Preparation

Hardware
own notebooks
VPN

Software
Java - version 7, hopefully v8 will come soon
IntelliJ IDEA or other IDE
we use IntelliJ Idea, for Spring development we reccomend Ultimate edition (paid)
Kong
in a form of Vagrant box
central OpenAPI team will provide
VirtualBox, latest version
Vagrant, latest version
Git
XMind, free edition will do (http://www.xmind.net/download/win/)
Node.js + NPM, version 6 and newer
all following tools will be installed automatically with base_project.zip
Dredd - API testing
Aglio - HTML documentation generation
Hercule - merging documentation
Drakov - mocking tool
Drafter - API Blueprint parser
(Dredd - API testing, Aglio - HTML documentation generation, Hercule - merging documentation, Drakov - mocking tool, Drafter - API Blueprint parser)
favorite text editor (API Blueprint), one of:
Sublime
Atom
VIm
any other, but former have API Blueprint syntax highlighting
Maven
Cucumber

Expected knowledge (at least some of those)
Java (ATL development)
Spring, Spring-REST - https://spring.io/guides/gs/rest-service/
REST APIs (in general)
http://www.restapitutorial.com/index.html
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api
OAuth2 - how it works
https://tools.ietf.org/html/rfc6749
https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2
DevOps
Git - https://githowto.com/
Jenkins - http://www.tutorialspoint.com/jenkins/
Maven - https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
Analysis
API Blueprint - https://apiblueprint.org/
MSON - https://apiblueprint.org/documentation/mson/tutorial.html
Design principles -
Testing
Dredd - https://dredd.readthedocs.io/en/latest/
Cucumber - https://cucumber.io/
Kong API gateway - basic principles - https://getkong.org
Open API architecture -

API Development Tools


ComponentProposed Tool(s)CommentsLink(s)RolesNext steps
Source Code RepositoryGitLabA distributed code versioning tool based on GIT with support of pull requests and code review will be set up.git.ibts.netEveryone from OpenAPI community should have r/w access to openapi repositoriesIDM account (OFFICE) is sufficient for RO access. RW access to OpenAPI project to be done manually by project administrator.
see also Versioning and branches
openapi/VNi-api - project for OpenAPI design & interfaces implementation
openapi/VNi-impl - project for OpenAPI ATL implemetnation
Build ToolJenkinsAn automated build and deployment tool.hudson.ibts.net (current)Everyone from OpenAPI community should have r/w access to openapi build jobsIDM account (OFFICE) for log in.
 jenkins.com.vn (since Feb 2017)Create new security group in office domain, which will be mapped to our project OT role. Based on this role community will have access only to openapi part of jenkins. Specific access to be prepared on jenkins by admin - Martin Kulich. We need to define what we need.
Local instance is recommended to speed things up and save some network traffic.
Component RepositoryNexusA SW component repository for distribution of ready Open API components (such as the API GW, Credential Store, Developer's Portal). nexus.com.vnEveryone from OpenAPI community should have read access, write access only for Jenkins jobs.IDM account is sufficient (for RO access log in is not needed).
Local instance is recommended to speed things up and save some network traffic.
Code Quality  SonarAn automated code quality check tool.sonar.com.vntbdVNG account is sufficient.
Container / VMDockerComponent deployment tool?tbd
Issue TrackingJIRASeparate API project in JIRA issue tracker that supports definition and control of:https://jira.ibts.net/jira/browse/APIEveryone from OpenAPI community has r/w access to API project.RV_APP_JIRA_GENERAL, API project, Executive user
Business Service – high-level business task (one item on the Open API - Business Service Stack)
EPIC, Story, Task (SubTask) – agile board for the planned local and central development
Bug – unplanned issue
Public read acces to all VNI employees (role RV_APP_JIRA_GENERAL). All Open API team members (including Country teams) will have read/write privilege assigned by Benda Jan (role Executive user).
Project DocumentswikiThis wiki API HomeEveryone from OpenAPI community has RW access to API wiki.RV_APP_WIKI01_API_RW
Each team member should have read access to it at least.
In case when edit privilege is required, contact Benda Jan (add role RV_APP_WIKI01_API_RW).
File ShareVN DepositBoxA web-based solution for exchange of large files (e.g. Vagrant boxes).https://VN-depositbox.ibts.net/Everyone with an VNG account should be able to log in and post/receive files. Anyone outside of VNG can drop a file for someone inside VNG.VNG account is sufficient.
VN-depositbox is a service to make it easy for you to move files, including large files up to 169.6 GB, in and out of EmbedIT.
Static API Documentationpublic webA www server hosting the public documentation (should be part of the Developer's portal?).?

OpenAPI Specification format comparison


CategoryTitleSwaggerRAMLAPI Blue Print
SummaryProsHigh adoption rate, good code-level tooling, large and active community of users and supporters, greater support for multiple languagesGood online design tools and mature supporting infrastructure, seemingly useful advanced constructs, human readable format, hight industry backingGood online design tools, good community involvement, easy to understand, simple to write
SummaryConsBottom-up, lacks advanced constructs for metadataVery new to the game, low adoption, lacks code-level tooling, still unproven long-termNew to the game, low adoption, lacks advanced constructs and code-level tooling, complex installation
BasicPrimary SponsorSmartBearMULESOFTAPIARY
BasicFormatJSONYAMLMARKDOWN
BasicInitiatives towards Industry Standard Open API Initiative (OAI) - Has announced an open governance model around the Swagger Specification under the Linux Foundation.Not AvailaibleNot Availaible
BasicOpen Source InititivesThe Swagger page claims  itself to be 100% open source software. Starting January 1st 2016 the Swagger Specification has been donated to to the Open API Initiative (OAI) and has been renamed to the OpenAPI Specification.RAML is backed by a large open source community providing hundreds of pre-built, customizable tools.API Blueprint is completely open sourced under the MIT license.
BasicIndustry BackingReverb, 3Scale, ApigeeMulesoft, SOA, AngularJS, Intuit, Box, PayPal, & Programmable WebApiary
BasicCurrent Version211A9
BasicWorkgroupYesYesNo
BasicInitial commitJuly 2011September 2013April 2013
BasicAPI Design ApproachBottom-upTop-downTop-down
CommunityCommunity BasedBy far has the largest comnmunity and has been around since 2011.Has gained lot of traction of late in the enterprises. RAML is backed by a large open source community providing hundreds of pre-built, customizable tools for all your RESTful API needsIs up and coming
CommunityStackoverflow questions3181288696
CommunityGitHub Stars37733913002
CommunityGitHub contributors1142638
CommunityGitHub whatchers276154147
CommunityGitHub forks2196118872
CommunityGoogle search (name + “rest”)733K85K350K
REST ModelingStub generatorsYesYesYes (Limited Languages).    
REST ModelingResourcesX (“api”)XX
REST ModelingMethods/ ActionsX (“operation”)X (“method”)X (“action”)
REST ModelingQuery ParametersXXX
REST ModelingPath/URL ParametersXXX
REST ModelingHeader ParametersXXX
REST ModelingRepresentations (status codes, mime-types)XXX
REST ModelingDocumentationXXX
REST ModelingAuthenticationBasic, API-Key, OAuth 2Basic, Digest, Oauth, 1&2
REST ModelingRepresentation MetadataJSON Schema (subset) (inline/ external) (inline)
REST ModelingNested ResourcesXX
REST ModelingComposition/ InheritanceTraits, Resource TypesResource Models
REST ModelingFile inclusionsX
REST ModelingAPI Version metadataXX
REST ModelingSample RepresentationsXX
ToolingLive Documentation DemoYesYesYes
ToolingAuthoring(3rd party)API-Designerapiary.io
ToolingAd-hoc testingSwagger-UIAPI-Consoleapiary.io
ToolingAPI ConsoleEvery HTTP Action Verb is separateGrouped by endpoint with drill down into verbsUnknown
ToolingAPI BuilderTools for most popular languagesMuleStudio APIKit (Visual, drag and drop), Osprey (Node)Node & .Net bindings
ToolingDocumentation BuilderSingle page documentation builderSingle or multiple page documentation builder (PHP only)Single or multiple page documentation builder
ToolingMocking(3rd party)XX
ToolingServer CodeJava, scala, ruby, node.jsJava(3rd party)
ToolingGenerate from codeJava, 3rd party (go, python)
ToolingLanguage SupportClojure, Go, JS, Java, .Net, Node, PHP, Python, Ruby, ScalaJS, Java, Node, PHP, Python, RubyNode, .Net
ToolingValidationXX (java)X
ToolingParsingJava, jsJava, jsC++ (nodejs, c#)
ToolingEasy of Getting StartedVery easy, everything in Github reposVery easy, everything in Github reposInstallation required