Friday, 6 December 2013

reset to one old commit on both local and remote computers

When using GitLab server, sometime I will reset to one old commit because I mess up in current commit.
On working computer, using this
git reset --hard 05a0753bd3e24922dd46bb6a04dc93dda9a9ec9f

Then try to push it to GitLab server, it doesn't work even if I push with --force option.
 remote: error: denying non-fast-forward refs/heads/develop (you should pull first)

Now the simplest way is to delete remote branch first, then push branch from local computer to GitLab server again.
git push origin :develop
git push origin develop

: means delete remote develop branch here.

Tuesday, 3 December 2013

Try valgrind

Today, I will try valgrind, a tool to analyze c++ app.
Download the newest version:
wget http://valgrind.org/downloads/valgrind-3.9.0.tar.bz2
unzip it
tar jxvf valgrind-3.9.0.tar.bz2
 then enter the folder and compile it with root account
cd valgrind-3.9.0
./configure
make
make install
install required libs
apt-get install libc6-dbg

Then use valgrind to launch your app, e.g.
# valgrind --leak-check=yes --log-file=profile ./data_service_d ./config.xml &
You will get the below from profile file in current folder.
[1] 25076
root@vm8:/home/dist/tcp# ==25076== Memcheck, a memory error detector
==25076== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==25076== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==25076== Command: ./data_service_d ./config.xml
==25076== 

use boost log step 4

Now, I apply the boost log to my one of existing applications to replace CppCMS log. I use it to do the following jobs, the size of one single log file is limited less than 10 MB, and the free disk space must be 3GB.
To use this, there are some issues need to be known.
1. include correct header files

#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/sinks/text_file_backend.hpp>
#include <boost/log/utility/setup/file.hpp>
#include <boost/log/utility/setup/common_attributes.hpp>
#include <boost/log/sources/severity_logger.hpp>
#include <boost/log/sources/record_ostream.hpp>
namespace logging = boost::log;
namespace src = boost::log::sources;
namespace sinks = boost::log::sinks;
namespace keywords = boost::log::keywords;
2. register_simple_formatter_factory is required, otherwise you cannot see severity field in log file
void InitLog() {
  boost::log::register_simple_formatter_factory< boost::log::trivial::severity_level, char >("Severity");
  logging::add_file_log(
keywords::file_name = AppHolder::Instance().config().log_folder + "/sign_%Y-%m-%d_%H-%M-%S.%N.log",
keywords::rotation_size = 10 * 1024 * 1024,
keywords::time_based_rotation = sinks::file::rotation_at_time_point(0, 0, 0),
keywords::format = "[%TimeStamp%] (%Severity%) : %Message%",
keywords::min_free_space=3 * 1024 * 1024
);
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::debug);
}

3.  In main function, set attributes after calling InitLog like this:
    InitLog();
    logging::add_common_attributes();
    using namespace logging::trivial;
    src::severity_logger< severity_level > lg;
    BOOST_LOG_SEV(lg, info) << "thread id: " << this_thread::get_id() << " Initialization succeeded";

Now, have a look at the log files, each file size is less than 10M, and one new file will be created at midnight.

 -rw-r--r-- 1 root root  10M Dec  3 23:16 sign_2013-12-03_23-00-01.0.log
-rw-r--r-- 1 root root  10M Dec  3 23:36 sign_2013-12-03_23-16-55.1.log
-rw-r--r-- 1 root root  10M Dec  3 23:55 sign_2013-12-03_23-36-21.2.log
-rw-r--r-- 1 root root 2.4M Dec  4 00:00 sign_2013-12-03_23-55-33.3.log
-rw-r--r-- 1 root root  10M Dec  4 00:19 sign_2013-12-04_00-00-00.4.log
-rw-r--r-- 1 root root  10M Dec  4 00:38 sign_2013-12-04_00-19-30.5.log
-rw-r--r-- 1 root root  10M Dec  4 00:58 sign_2013-12-04_00-38-48.6.log
-rw-r--r-- 1 root root  10M Dec  4 01:17 sign_2013-12-04_00-58-06.7.log
-rw-r--r-- 1 root root  10M Dec  4 01:36 sign_2013-12-04_01-17-19.8.log
-rw-r--r-- 1 root root  10M Dec  4 01:56 sign_2013-12-04_01-36-45.9.log
-rw-r--r-- 1 root root  10M Dec  4 02:15 sign_2013-12-04_01-56-07.10.log

Look inside one log file:
[2013-Dec-04 10:17:45.728393] (debug) : object id: 529df217186f983f62ffa718 sent data: 01 30 30 30 43 44 31 4F 4B 31 30 03  sent size:12

4.  In other .cc files, just include one header file
 #include <boost/log/trivial.hpp>
And use BOOST_LOG_TRIVIAL() MACROS
BOOST_LOG_TRIVIAL(debug) << "object id: " << id_ << " sent data: " << PrintBytesAsHexString(*data, data->size()) << " sent size:" << data->size();

That's enough.
Inside boost log, there are many details need to be learned. I have not time for now to analyze it. Will write blogs about this when free in the future.

use boost log step 3

Now, I need to write log to local file. My example code is simpler than official doc.

#include <iostream>
#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/utility/setup/file.hpp>
namespace logging = boost::log;
using namespace std;
void SetFilter1() {
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::info);
}
void SetFilter2() {
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::debug);
}
int main () {
  cout << "hello, world" << endl;
  logging::add_file_log("sample.log");
  SetFilter1();
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
 
  BOOST_LOG_TRIVIAL(info) << "--------------------" << endl;
  SetFilter2();
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
}
Do not forget to include <boost/log/utility/setup/file.hpp>, otherwise you will get compilation error like so:
no member named 'add_file_log' in namespace 'boost::log'

 Now, run the app, you will see sample file is created in current folder.
$ cat sample.log
An informational severity message
A warning severity message
An error severity message
A fatal severity message
--------------------
A debug severity message
An informational severity message
A warning severity message
An error severity message
A fatal severity message
Here we encounter a concept "sink", sink decides where the log info will be written. In the previous two articles, I didn't add sink. The boost log uses console as default sink. Now, because I call add_file_log function, so the default sink is replaced by the file sink now. To setup sink, you need to add it into core. Just one statement in above example:
logging::add_file_log("sample.log");
boost log has another concept-- source. This doc describes the architecture of boost log design.


The left side is log source, it collects the log info in our app. just like this statement
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
The right side is sink, it processes log info, decides where the log should be put, how to use it.
And the logging core interconnects them. 

Monday, 2 December 2013

Use boost log step 2

This time, I will try filter. first code from official doc,

#include <iostream>
#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
namespace logging = boost::log;
using namespace std;
void SetFilter() {
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::info);
}
int main () {
  cout << "hello, world" << endl;
  SetFilter();
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
}

When run the binary, output is changed, trac and debug level info were filtered out.
$ ./main
hello, world
[2013-12-03 15:34:47.343223] [0x000007f680e76774] [info]    An informational severity message
[2013-12-03 15:34:47.343324] [0x000007f680e76774] [warning] A warning severity message
[2013-12-03 15:34:47.343341] [0x000007f680e76774] [error]   An error severity message
[2013-12-03 15:34:47.343356] [0x000007f680e76774] [fatal]   A fatal severity message

But could we change the filter in runtime. Let's try. Change the code now:

#include <iostream>
#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
namespace logging = boost::log;
using namespace std;
void SetFilter1() {
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::info);
}
void SetFilter2() {
  logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::debug);
}
int main () {
  cout << "hello, world" << endl;
  SetFilter1();
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
 
  cout << "--------------------" << endl;
  SetFilter2();
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
}

Ok, watch the result now.

$ ./main
hello, world
[2013-12-03 15:37:54.399513] [0x000007fd7709a374] [info]    An informational severity message
[2013-12-03 15:37:54.399612] [0x000007fd7709a374] [warning] A warning severity message
[2013-12-03 15:37:54.399630] [0x000007fd7709a374] [error]   An error severity message
[2013-12-03 15:37:54.399644] [0x000007fd7709a374] [fatal]   A fatal severity message
--------------------
[2013-12-03 15:37:54.399666] [0x000007fd7709a374] [debug]   A debug severity message
[2013-12-03 15:37:54.399680] [0x000007fd7709a374] [info]    An informational severity message
[2013-12-03 15:37:54.399693] [0x000007fd7709a374] [warning] A warning severity message
[2013-12-03 15:37:54.399706] [0x000007fd7709a374] [error]   An error severity message
[2013-12-03 15:37:54.399719] [0x000007fd7709a374] [fatal]   A fatal severity message

It works. That means we can change the log filter without redeploying app. In my TCP server, it can listen to another port for admin usage and receive special message for changing log filter.

Good feature! I like this.



Use boost log step1

I get used to log library from CppCMS, it's simple. Boost releases the official log library since 1.54 version. After reading some doc, I decide to change to boost::log because the following reasons:
1. There are about 2,000 devices which connect to my TCP server, the TCP server saves all detail information. For example, when the device logs into the server, when it quits, what message is sent by device or server. And the amount of devices will increase in the future. Thus cause a few very large log files are created.

2. We will analyze the log file for profiling the problems of TCP server, also find out how our customers use their devices. When a problem occurs, administrator should receive waning email. And the product people could see the user behavior analysis report on web page to help them improve our product.

Obviously, boost new log library is designed to do the above jobs. That's what I expected.

Here is a example, most of code from boost log document:
http://www.boost.org/doc/libs/1_55_0/libs/log/doc/html/log/tutorial.html#log.tutorial.trivial

I am using Ubuntu 13.10 installed clang 3.4, also I built boost 1.55 libraries with clang & c++11 option.
Refer to my build command below:

./bootstrap.sh --with-libraries=system,filesystem,log,thread --with-toolset=clang
./b2 toolset=clang cxxflags="-std=c++11"

Now, create a main.cc file, write some code like this:
#include <iostream>
#include <boost/log/trivial.hpp>

using namespace std;

int main () {
  cout << "hello, world" << endl;
  BOOST_LOG_TRIVIAL(trace) << "A trace severity message";
  BOOST_LOG_TRIVIAL(debug) << "A debug severity message";
  BOOST_LOG_TRIVIAL(info) << "An informational severity message";
  BOOST_LOG_TRIVIAL(warning) << "A warning severity message";
  BOOST_LOG_TRIVIAL(error) << "An error severity message";
  BOOST_LOG_TRIVIAL(fatal) << "A fatal severity message";
}

Compile code and link it to binary:
c++ -g -std=c++11 -Wall -DBOOST_LOG_DYN_LINK -o ./main.o -c ./main.cc
c++ -g -std=c++11 ./main.o -o main -rdynamic -lpthread -lboost_log -lboost_system -lboost_thread -lboost_filesystem

Now, execute ./main binary, it outputs:
$ ./main
hello, world
[2013-12-03 14:35:14.309093] [0x000007f362c17d74] [trace]   A trace severity message
[2013-12-03 14:35:14.309208] [0x000007f362c17d74] [debug]   A debug severity message
[2013-12-03 14:35:14.309224] [0x000007f362c17d74] [info]    An informational severity message
[2013-12-03 14:35:14.309237] [0x000007f362c17d74] [warning] A warning severity message
[2013-12-03 14:35:14.309250] [0x000007f362c17d74] [error]   An error severity message
[2013-12-03 14:35:14.309262] [0x000007f362c17d74] [fatal]   A fatal severity message


By default, the BOOST_LOG_TRIVIAL outputs all information to console. Some people ran into the problem
 undefined reference to `boost::log::v2s_mt_posix::trivial::logger::get()'

Add -DBOOST_LOG_DYN_LINK to solve this.


Saturday, 1 September 2012

Try protojs serials : C. return protocol buffer from web service

I will implement a web app using CppCMS framework. Yes, it's a C++ framework.
The basic idea is to generate the binary stream, send it via response to browser, set the content type to
application/x-protobuf.

This post doesn't describe the usage of CppCMS. Please refer to cppcms.com if needed.

In web service constructor, map the url request to test_group member function.

dispatcher().assign("/testGroup" , &kaimei_web_site::test_group , this);

Some part  of test_group codes list here:


void kaimei_web_site::test_group() {
  kaimei::response res;
  try {
    ....
    response().content_type("application/x-protobuf");
    freebird::group g;
    g.set_id("1");
    g.set_name("a");
    g.set_description("d");
    g.set_device_number("5");
    g.SerializePartialToOstream(&response().out());

  } catch(std::exception const& ex) {
    ...
  }
}

That's all. Now launch browser, access the web site using /testGroup path. Browser will download a file called testGroup.


Unfortunately, the protojs can't parse the stream transferred from server.
Chrome always says:ArrayBuffer values are deprecated ...

I will update this post later.


Followers

Contributors