마이닝 서버 설치
FlowKat에 수집된 XLOG 데이터를 RAW 레벨로 제공할 수 있는 서버를 설치합니다.
마이닝 서버 설정 예제
마이닝 서버와의 연동을 위해 수집 서버의 환경 변수를 수정합니다.
수집 서버 환경 설정 파일 위치로 이동
/app/flowkat/collect.server/bin/setenv.sh
$ cd /app/flowkat/collect.server/bin/
$ vi setenv.sh
...
# 서버 플러그인 XLOG 활성화 여부
export server_plugin_xlog_enabled=true
# 서버 플러그인 XLOG 최소 보관 기간
export server_plugin_xlog_dump_duration_day=7
# 서버 플러그인 XLOG 파일 덤프 저장 폴더
export server_plugin_xlog_dump_root_dir=${FLOWKAT_COLLECTOR_SERVER_HOME}/filedump
# 서버 플러그인 XLOG 파일 덤프 이력 폴더
export server_plugin_xlog_dump_rotate_dir=${FLOWKAT_COLLECTOR_SERVER_HOME}/filedmup/history
${FLOWKAT_HOME}/collect.server/flowkat.conf 파일에 위 파라미터를 설정하면 파일 기준으로 동작합니다. 다만, 환경 변수 ${FLOWKAT_COLLECTOR_SERVER_HOME} 값은 지원되지 않습니다
환경 변수 파일 설정
마이닝 서버 환경 설정 파일 위치로 이동
/app/flowkat/mining.server/container/.env
$ cd /app/flowkat/mining.server/container/.env
$ vi .env
# 마이닝 서버 이미지 버전
MINING_VERSION=1.0.1
# 마이닝 서버에서 읽을 로그 위치 설정
MINING_PATH=/app/collect.server/filedump
#PostgreSQL 저장소
PG_DATA_PATH=./postgres/PGDATA
#PostgresSQL DB 패스워드
PG_PASSWORD='A4Fu@*BViO$cj'
# 최초 DB 파티션 설정시, 날짜별 파티션 테이블 생성 기준일을 설정
MINING_INIT_PARTITION_DAYS=31
# 최초 DB 파티션 설정시, 파티션 최대 보존 기간을 설정
MINING_INIT_PARTITION_KEEP_MONTH=12
MINING_PATH 경로는 수집 서버의 XLOG 플러그인 설정 경로와 반드시 일치해야 합니다.
대시보드 서버와 마이닝 서버 연동
마이닝 서버를 플로우캣 대시보드에서 확인할 수 있는 설정입니다.
/app/flowkat/dashboard.server/bin/setenv.sh
$ cd /app/flowkat/dashboard.server/bin/
$ vi setenv.sh
...
# 트랜잭션 마이닝 메뉴 활성화 여부
export FLOWKAT_TX_MINING_ENABLED=true
# 트랜잭션 마이닝 서버 연동 주소
export FLOWKAT_TX_MINING_URL=http://127.0.0.1:8086
# 트랜잭션 마이닝 배치 처리 워크 스레드 개수
export FLOWKAT_TX_MINING_WORKER_COUNT=2
# 트랜잭션 마이닝 배치 저장소 디렉토리
export FLOWKAT_TX_SAVE_DIR=$FLOWKAT_DASHBOARD_SERVER_HOME/filedump
파일 설정 완료 후 대시보드 서버 기동
/app/flowkat/dashboard.server/bin
$ cd /app/flowkat/dashboard.server/bin
$ ./startup.sh
_____ _ _ __ _
| ___| | _____ _| |/ /__ _| |_
| |_ | |/ _ \ \ /\ / / ' // _` | __|
| _| | | (_) \ V V /| . \ (_| | |_
|_| |_|\___/ \_/\_/ |_|\_\__,_|\__|
FlowKat Monitoring Solution
FlowKat version 5.0.0
2023-06-09 10:33:25.335 INFO 11735 --- [ main] flowkat.paper.extension.Application : Starting Application using Java 1.8.0_301 on blueberry with PID 11735 (/home/kranian/project/flowkat-ex/build/classes/java/main started by kranian in /home/kranian/project/flowkat-ex)
2023-06-09 10:33:25.337 INFO 11735 --- [ main] flowkat.paper.extension.Application : The following 1 profile is active: "prod"
20230609 10:33:25 [A103] HTTP javax/servlet/http/HttpServlet
....
.security.web.authentication.logout.LogoutFilter@33502cfe, flowkat.paper.extension.service.provider.security.filter.JwtFilter@2aeb3f3b, flowkat.paper.extension.config.SpaWebFilter@281028ce, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@2487b621, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@3d2ff73a, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@46ef7340, org.springframework.security.web.session.SessionManagementFilter@5f01361e, org.springframework.security.web.access.ExceptionTranslationFilter@2dc77962, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@401d83ce]
2023-06-09 10:33:29.657 INFO 11735 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8081 (http) with context path ''
2023-06-09 10:33:29.667 INFO 11735 --- [ main] flowkat.paper.extension.Application : Started Application in 4.801 seconds (JVM running for 8.734)
마이닝 서버 기동
파일 설정 완료 후 기동
/app/flowkat/mining.server/container
$ cd /app/flowkat/mining.server/container
$ sudo docker compose up -d
[+] Running 4/4
✔ Container postgres Started 0.4s
✔ Container mining Started 0.9s
✔ Container logstash Started 1.0s
✔ Container filebeat Started 1.4s
컨테이너 기동 확인
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f459bd51cfa7 flowkat/filebeat-oss:7.10.1 "/usr/local/bin/dock…" 9 seconds ago Up 8 seconds filebeat
6601e6c6ebcc flowkat/trace.log.server:1.0.0 "sh -c 'java ${JAVA_…" 9 seconds ago Up 8 seconds 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp mining
61f393300cd8 flowkat/logstash-jdbc-oss:7.10.1 "/usr/local/bin/dock…" 9 seconds ago Up 8 seconds 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 9600/tcp logstash
4ff5f92ba4aa flowkat/postgres-partman:16.5 "/opt/bitnami/script…" 58 minutes ago Up 8 seconds 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp postgres
마이닝 서버 로그 확인
$ sudo docker compose logs
postgres | postgresql 11:53:30.58 INFO ==> Validating settings in POSTGRESQL_* env vars..
flowkat.mining | FlowKat Mining server
flowkat.mining | Mining version 1.0.0
flowkat.mining |
flowkat.mining |
flowkat.mining | 2024-08-08T12:51:45.595+09:00 INFO 1 --- [ main] flowkat.tx.analyze.Application : Starting Application v1.0.0 using Java 17.0.12 with PID 1 (/app/flowkat/mining.server/trace.log.server.jar started by flowkat in /app/flowkat)
flowkat.mining | 2024-08-08T12:51:45.618+09:00 INFO 1 --- [ main] flowkat.tx.analyze.Application : The following 1 profile is active: "local"
flowkat.mining | 2024-08-08T12:51:48.720+09:00 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
flowkat.mining | 2024-08-08T12:51:49.489+09:00 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 731 ms. Found 1 JPA repository interface.
flowkat.mining | 2024-08-08T12:51:53.595+09:00 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port 8086 (http)
flowkat.mining | 2024-08-08T12:51:53.656+09:00 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
flowkat.mining | 2024-08-08T12:51:53.667+09:00 INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.18]
flowkat.mining | 2024-08-08T12:51:53.990+09:00 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
flowkat.mining | 2024-08-08T12:51:54.007+09:00 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 8069 ms
flowkat.mining | 2024-08-08T12:51:55.483+09:00 INFO 1 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
flowkat.mining | 2024-08-08T12:51:55.729+09:00 INFO 1 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 6.4.1.Final
flowkat.mining | 2024-08-08T12:51:55.879+09:00 INFO 1 --- [ main] o.h.c.internal.RegionFactoryInitiator : HHH000026: Second-level cache disabled
logstash | Using bundled JDK: /usr/share/logstash/jdk
logstash | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ubuntu@flowkat:~/flowkat/mining.server/container$ sudo docker compose logs -f
postgres | postgresql 11:53:30.48 INFO ==>
postgres | postgresql 11:53:30.49 INFO ==> Welcome to the Bitnami postgresql container
postgres | postgresql 11:53:30.49 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgres | postgresql 11:53:30.50 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgres | postgresql 11:53:30.50 INFO ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
postgres | postgresql 11:53:30.51 INFO ==>
postgres | postgresql 11:53:30.52 INFO ==> ** Starting PostgreSQL setup **
postgres | postgresql 11:53:30.58 INFO ==> Validating settings in POSTGRESQL_* env vars..
postgres | postgresql 11:53:30.70 INFO ==> Loading custom pre-init scripts...
postgres | postgresql 11:53:30.71 INFO ==> Initializing PostgreSQL database...
postgres | postgresql 11:53:30.73 INFO ==> pg_hba.conf file not detected. Generating it...
postgres | postgresql 11:53:30.74 INFO ==> Generating local authentication configuration
postgres | postgresql 11:53:32.65 INFO ==> Starting PostgreSQL in background...
postgres | postgresql 11:53:32.82 INFO ==> Changing password of postgres
postgres | postgresql 11:53:32.92 INFO ==> Configuring replication parameters
postgres | postgresql 11:53:32.98 INFO ==> Configuring synchronous_replication
postgres | postgresql 11:53:32.98 INFO ==> Configuring fsync
postgres | postgresql 11:53:33.03 INFO ==> Stopping PostgreSQL...
postgres | waiting for server to shut down.... done
postgres | server stopped
postgres | postgresql 11:53:33.15 INFO ==> Loading custom scripts...
postgres | postgresql 11:53:33.16 INFO ==> Enabling remote connections
postgres | postgresql 11:53:33.18 INFO ==> ** PostgreSQL setup finished! **
postgres |
postgres | postgresql 11:53:33.23 INFO ==> ** Starting PostgreSQL **
postgres | 2024-08-08 02:53:33.291 GMT [1] LOG: pgaudit extension initialized
마이닝 서버 서비스 확인
docker ps
...
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
187f72cfa86a flowkat/trace.log.server:1.0.2 "sh -c 'java ${JAVA_…" 19 seconds ago Up 18 seconds 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp mining
d46c8a9a5737 flowkat/logstash-jdbc-oss:7.10.1 "/usr/local/bin/dock…" 19 seconds ago Up 18 seconds 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 9600/tcp logstash
4fcf302a8bfa flowkat/postgres-partman:16.5 "/opt/bitnami/script…" 19 seconds ago Up 18 seconds 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp postgres
...
트랜잭션 마이닝 메뉴 활성화
설정 > 사용자 관리 > 권한메뉴에서 admin 권한을 선택합니다.
설정 완료 후 트랜잭션 마이닝 메뉴를 확인하기 위해 새로고침이 필요할 수 있습니다.
화면 설정 > 분석 > 트랜잭션 마이닝체크박스를 선택하고 저장합니다.
트랜잭션 마이닝 활성화 확인
- 배치 저장소의 정보를 확인합니다.
- 데이터베이스의 정보를 확인합니다.
마이닝 서버 중지
마이닝 서버 실행 파일 위치로 이동
/app/flowkat/mining.server/container
$ cd /app/flowkat/mining.server/container
마이닝 서버 실행 파일 위치에서 중지 명령어를 실행
$ sudo docker compose down
[+] Running 4/4
✔ Container mining Removed 0.3s
✔ Container logstash Removed 7.5s
✔ Container postgres Removed 0.2s
✔ Network container_default Removed 0.2s