Browse Source

First jenkins build

pull/1/head
Michael COSTA 6 months ago
parent
commit
af1e2c8105
17 changed files with 286 additions and 2396 deletions
  1. +5
    -3
      .gitignore
  2. +0
    -708
      CHANGELOG.md
  3. +1
    -1
      CONTRIBUTING.md
  4. +3
    -2
      Jenkinsfile
  5. +17
    -8
      Makefile
  6. +95
    -1562
      README.md
  7. +0
    -18
      requirements.txt
  8. BIN
      scrippy-remote.png
  9. +7
    -0
      scrippy_remote/__init__.py
  10. +4
    -5
      scrippy_remote/remote/__init__.py
  11. +39
    -31
      scrippy_remote/remote/ftp.py
  12. +2
    -3
      scrippy_remote/remote/gncftp.py
  13. +50
    -33
      scrippy_remote/remote/ssh.py
  14. +4
    -20
      setup.cfg
  15. +2
    -2
      sonar-project.properties
  16. +7
    -0
      tests/__init__.py
  17. +50
    -0
      tests/test_remote_ssh.py

+ 5
- 3
.gitignore View File

@ -1,13 +1,15 @@
*un~
*.swp
*.pyc
*/__pycache__
**/__pycache__/
build/
dist/
infra.egg-info/
python_infra.egg-info/
scrippy_remote.egg-info/
test.py
populate_dir.py
.vscode/
venv/
.pytest_cache/
.idea
.coverage
junit_report.xml

+ 0
- 708
CHANGELOG.md View File

@ -1,708 +0,0 @@
# Changelog
Fichier généré automatiquement par auto-changelog
## 1.1.52 (2021-01-20)
#### Fixes
* Makefile: Bad VERSION number
* Makefile: Bad VERSION number
* Makefile: Bad VERSION number
* bumpversion current_version
* Makefile: More targets
* Makefile: More targets
* Makefile: More targets
* Makefile: More targets
* Makefile: More targets
* Makefile
#### Others
* Makefile bumpversion and release targets
* Makefile bumpversion and release targets
Full set of changes: [`1.1.46...1.1.52`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.46...1.1.52)
## 1.1.46 (2020-12-29)
#### Fixes
* remote.ssh.exec_command should not show command unless --debug is used
Full set of changes: [`1.1.45...1.1.46`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.45...1.1.46)
## 1.1.45 (2020-12-29)
#### Fixes
* remote.ssh.exec_command should not show command unless --debug is used
Full set of changes: [`1.1.44...1.1.45`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.44...1.1.45)
## 1.1.44 (2020-12-09)
#### Fixes
* TERM is not defined on remote execution
Full set of changes: [`1.1.43...1.1.44`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.43...1.1.44)
## 1.1.43 (2020-12-09)
#### Fixes
* No more color in Jobscheduler + colors in log files
* No more color in Jobscheduler + colors in log files
* No more color in Jobscheduler + colors in log files
Full set of changes: [`1.1.42...1.1.43`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.42...1.1.43)
## 1.1.42 (2020-11-25)
#### Fixes
* SIN-15126
Full set of changes: [`1.1.41...1.1.42`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.41...1.1.42)
## 1.1.41 (2020-11-25)
#### Fixes
* SIN-15126
* SIN-15126
* SIN-15126
Full set of changes: [`1.1.40...1.1.41`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.40...1.1.41)
## 1.1.40 (2020-11-25)
#### Fixes
* SIN-15126
* SIN-15126
Full set of changes: [`1.1.39...1.1.40`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.39...1.1.40)
## 1.1.39 (2020-10-27)
#### Fixes
* (errorhandler): correction gestion du sys.exit
Full set of changes: [`1.1.38...1.1.39`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.38...1.1.39)
## 1.1.38 (2020-10-27)
#### New Features
* (history): en cas d'exception affiche le nom de l'exception dans le hist à la place de l'exit_code
* (error_handler): ajout gestion d'exception
#### Fixes
* commit only if repo is dirty
Full set of changes: [`1.1.37...1.1.38`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.37...1.1.38)
## 1.1.37 (2020-10-26)
#### Fixes
* infra.git: Utilisation de l objet Repo.git
* infra.git: Prise en charge des suppressions de fichiers
* infra.git: Prise en charge des suppressions de fichiers
* infra.git: Prise en charge des suppressions de fichiers
Full set of changes: [`1.1.36...1.1.37`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.36...1.1.37)
## 1.1.36 (2020-10-26)
#### Fixes
* api.validator global catch
* infra.api: There is NO Validator class
* infra.api: Validator is available from infra.api
* api: flake8 ready
Full set of changes: [`1.1.35...1.1.36`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.35...1.1.36)
## 1.1.35 (2020-10-23)
#### New Features
* Added multi_options support to infra.git
Full set of changes: [`1.1.34...1.1.35`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.34...1.1.35)
## 1.1.34 (2020-10-23)
#### New Features
* infra.git: Gestion des dépôts Git
Full set of changes: [`1.1.33...1.1.34`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.33...1.1.34)
## 1.1.33 (2020-10-20)
#### Fixes
* remote.ssh: Secure regexp match + unshadowing dir builtin
Full set of changes: [`1.1.32...1.1.33`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.32...1.1.33)
## 1.1.32 (2020-10-19)
#### Fixes
* Lower remote.ssh default verbosity
#### Others
* SIN-13730 test execution sur le master
Full set of changes: [`1.1.31...1.1.32`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.31...1.1.32)
## 1.1.31 (2020-10-19)
#### Fixes
* Le controle de la conf se fait dans tous les cas
Full set of changes: [`1.1.30...1.1.31`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.30...1.1.31)
## 1.1.30 (2020-10-19)
#### Fixes
* (remote): Ssh exec_command correction bug commande qui ne se termine pas
Full set of changes: [`1.1.29...1.1.30`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.29...1.1.30)
## 1.1.29 (2020-10-16)
#### Fixes
* (remote): Ssh exec_command correction bug commande qui ne se termine pas
Full set of changes: [`1.1.28...1.1.29`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.28...1.1.29)
## 1.1.28 (2020-10-13)
Full set of changes: [`1.1.27...1.1.28`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.27...1.1.28)
## 1.1.27 (2020-10-13)
#### Fixes
* remote.ftp.put_file remove remote_dir first char only if needed
Full set of changes: [`1.1.26...1.1.27`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.26...1.1.27)
## 1.1.26 (2020-10-13)
#### Fixes
* remote.ftp.put_file remove remote_dir first char only if needed
Full set of changes: [`1.1.25...1.1.26`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.25...1.1.26)
## 1.1.25 (2020-10-13)
#### Fixes
* remote.ftp prot_c fallback to prot p
Full set of changes: [`1.1.24...1.1.25`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.24...1.1.25)
## 1.1.24 (2020-10-13)
#### Fixes
* remote.ftp prot_c fallback to prot p
Full set of changes: [`1.1.23...1.1.24`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.23...1.1.24)
## 1.1.23 (2020-10-13)
#### Fixes
* remote.ftp prot_c fallback to prot p
Full set of changes: [`1.1.22...1.1.23`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.22...1.1.23)
## 1.1.22 (2020-10-12)
#### Fixes
* (remote): Ssh exec_command correction bug commande qui ne se termine pas
Full set of changes: [`1.1.21...1.1.22`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.21...1.1.22)
## 1.1.21 (2020-10-06)
#### New Features
* (ssh): exec_command accept maintenant l'ensemble des paramètres de la methode de paramiko
* (ssh): exec_command changement du retour stdin -> exit_code voir readme https://gitlab-infra.ref.gnc/python-infra/python-infra/blob/master/README.md\[#ex](https://gitlab-infra.ref.gnc/python-infra/python-infra/issues/ex)%C3%A9cuter-une-commande-sur-un-h%C3%B4te-distant
Full set of changes: [`1.1.20...1.1.21`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.20...1.1.21)
## 1.1.20 (2020-10-02)
#### New Features
* (remote): remote Ssh ajout de la method sftp_stat
Full set of changes: [`1.1.19...1.1.20`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.19...1.1.20)
## 1.1.19 (2020-10-01)
Full set of changes: [`1.1.18...1.1.19`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.18...1.1.19)
## 1.1.18 (2020-10-01)
#### Fixes
* Check authorized user before using PIDStack
#### Others
* remote/gncftp: clean lint
Full set of changes: [`1.1.17...1.1.18`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.17...1.1.18)
## 1.1.17 (2020-10-01)
Full set of changes: [`1.1.16...1.1.17`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.16...1.1.17)
## 1.1.16 (2020-10-01)
#### New Features
* ajout des 'with' sur les class Ssh, DB, SpamAssassinClient, Ftp, GncFtp
#### Others
* correction gestion des tags
* suppression de la notification des Merge Request [ci-skip]
Full set of changes: [`1.1.15...1.1.16`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.15...1.1.16)
## 1.1.15 (2020-09-21)
Full set of changes: [`1.1.14...1.1.15`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.14...1.1.15)
## 1.1.14 (2020-09-17)
#### Fixes
* _find_remote_files: exit_on_error
#### Others
* Avertissement de non-confidentialité
Full set of changes: [`1.1.13...1.1.14`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.13...1.1.14)
## 1.1.13 (2020-09-16)
Full set of changes: [`1.1.12...1.1.13`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.12...1.1.13)
## 1.1.12 (2020-09-14)
#### Fixes
* (context): correction erreur sur le context en utilisant les 2 options --nolog et --no-log-file
Full set of changes: [`1.1.11...1.1.12`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.11...1.1.12)
## 1.1.11 (2020-08-24)
Full set of changes: [`1.1.10...1.1.11`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.10...1.1.11)
## 1.1.10 (2020-08-24)
Full set of changes: [`1.1.9...1.1.10`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.9...1.1.10)
## 1.1.9 (2020-08-24)
#### Fixes
* security fix + sonar quality fixes
* Jinja2 autoescape == True by default
Full set of changes: [`1.1.8...1.1.9`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.8...1.1.9)
## 1.1.8 (2020-08-24)
#### New Features
* stack_file_path extension
Full set of changes: [`1.1.7...1.1.8`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.7...1.1.8)
## 1.1.7 (2020-08-24)
#### Fixes
* PermissionError handling on lockfile
Full set of changes: [`1.1.6...1.1.7`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.6...1.1.7)
## 1.1.6 (2020-08-21)
#### Refactorings
* Better filelock logging handling
Full set of changes: [`1.1.5...1.1.6`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.5...1.1.6)
## 1.1.5 (2020-08-21)
#### Refactorings
* Better filelock logging handling
Full set of changes: [`1.1.4...1.1.5`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.4...1.1.5)
## 1.1.4 (2020-08-20)
Full set of changes: [`1.1.3...1.1.4`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.3...1.1.4)
## 1.1.3 (2020-08-20)
#### New Features
* max_instance
Full set of changes: [`1.1.2...1.1.3`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.2...1.1.3)
## 1.1.2 (2020-08-17)
Full set of changes: [`1.1.1...1.1.2`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.1.1...1.1.2)
## 1.1.1 (2020-08-17)
#### Refactorings
* api.apiloader
Full set of changes: [`1.0.153...1.1.1`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.153...1.1.1)
## 1.0.153 (2020-08-13)
#### Others
* auto-changlog fix diff url
* upgrade auto-changelog 0.4.0 -> 0.5.1
Full set of changes: [`1.0.152...1.0.153`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.152...1.0.153)
## 1.0.152 (2020-08-10)
Full set of changes: [`1.0.151...1.0.152`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.151...1.0.152)
## 1.0.151 (2020-08-07)
#### Refactorings
* VWMareApi
Full set of changes: [`1.0.150...1.0.151`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.150...1.0.151)
## 1.0.150 (2020-08-03)
#### New Features
* api.Client ajout SSL verify
Full set of changes: [`1.0.149...1.0.150`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.149...1.0.150)
## 1.0.149 (2020-08-03)
#### Fixes
* api.Client response referenced before assignement
Full set of changes: [`1.0.148...1.0.149`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.148...1.0.149)
## 1.0.148 (2020-08-03)
#### Others
* fix project url in setup.cfg
Full set of changes: [`1.0.147...1.0.148`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.147...1.0.148)
## 1.0.147 (2020-08-03)
#### New Features
* infra.api.Client
#### Fixes
* Better README.md
* Typo in README.md
* better doc
* grammar in README.dm
#### Others
* updated module requirements + added requirements.txt
Full set of changes: [`1.0.145...1.0.147`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.145...1.0.147)
## 1.0.145 (2020-07-30)
#### Others
* clean Jenkinsfile
Full set of changes: [`1.0.144...1.0.145`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.144...1.0.145)
## 1.0.144 (2020-07-30)
#### New Features
* notifyMergeRequest
Full set of changes: [`1.0.143...1.0.144`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.143...1.0.144)
## 1.0.143 (2020-07-30)
#### Docs
* Correction de la doc d install
Full set of changes: [`1.0.142...1.0.143`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.142...1.0.143)
## 1.0.142 (2020-07-27)
#### Fixes
* remote.gncftp.close
Full set of changes: [`1.0.141...1.0.142`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.141...1.0.142)
## 1.0.141 (2020-07-27)
#### New Features
* infra.conf.Config.get_section
Full set of changes: [`1.0.140...1.0.141`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.140...1.0.141)
## 1.0.140 (2020-07-16)
#### Fixes
* README.md: Ajoute doc sur nolog
Full set of changes: [`1.0.139...1.0.140`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.139...1.0.140)
## 1.0.139 (2020-07-16)
#### Fixes
* README.md: Ajoute doc sur nolog
Full set of changes: [`1.0.138...1.0.139`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.138...1.0.139)
## 1.0.138 (2020-07-16)
#### Fixes
* infra.db: type == postgres
Full set of changes: [`1.0.137...1.0.138`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.137...1.0.138)
## 1.0.137 (2020-07-08)
Full set of changes: [`1.0.136...1.0.137`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.136...1.0.137)
## 1.0.136 (2020-07-08)
#### Fixes
* python3.4 argparse.ArgumentParser has no allow_abbrev argument
Full set of changes: [`1.0.135...1.0.136`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.135...1.0.136)
## 1.0.135 (2020-06-22)
#### New Features
* infra.remote.Ssh.sftp_list
Full set of changes: [`1.0.134...1.0.135`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.134...1.0.135)
## 1.0.134 (2020-06-22)
#### Fixes
* sftp_delete must open sftp connection first
Full set of changes: [`1.0.133...1.0.134`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.133...1.0.134)
## 1.0.133 (2020-06-22)
#### New Features
* infra.remote.Ssh.sftp_delete
Full set of changes: [`1.0.132...1.0.133`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.132...1.0.133)
## 1.0.132 (2020-06-19)
#### New Features
* Ajout de la classe Db_from_cc
Full set of changes: [`1.0.131...1.0.132`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.131...1.0.132)
## 1.0.131 (2020-06-19)
#### Refactorings
* infra.remote.ssh
Full set of changes: [`1.0.130...1.0.131`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.130...1.0.131)
## 1.0.130 (2020-06-12)
#### Fixes
* Db.execute default params
* Db.execute default params
Full set of changes: [`1.0.129...1.0.130`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.129...1.0.130)
## 1.0.129 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.128...1.0.129`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.128...1.0.129)
## 1.0.128 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.127...1.0.128`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.127...1.0.128)
## 1.0.127 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.126...1.0.127`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.126...1.0.127)
## 1.0.126 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.125...1.0.126`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.125...1.0.126)
## 1.0.125 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.124...1.0.125`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.124...1.0.125)
## 1.0.124 (2020-06-12)
#### Fixes
* Db.execute default params
Full set of changes: [`1.0.123...1.0.124`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.123...1.0.124)
## 1.0.123 (2020-06-12)
#### Fixes
* Db.execute for Oracle
Full set of changes: [`1.0.122...1.0.123`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.122...1.0.123)
## 1.0.122 (2020-06-12)
#### Others
* Updated Readme for Oracle support
Full set of changes: [`1.0.121...1.0.122`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.121...1.0.122)
## 1.0.121 (2020-06-12)
#### Others
* ajustement ordre bump2version tag auto-changelog
Full set of changes: [`1.0.120...1.0.121`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.120...1.0.121)
## 1.0.120 (2020-06-12)
#### Fixes
* Typo in Jenkinsfile
Full set of changes: [`1.0.119...1.0.120`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.119...1.0.120)
## 1.0.119 (2020-06-12)
#### Others
* end of test
Full set of changes: [`1.0.118...1.0.119`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.118...1.0.119)
## 1.0.118 (2020-06-12)
#### Fixes
* default db_type is pgsql
#### Others
* This is a test
Full set of changes: [`1.0.117...1.0.118`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.117...1.0.118)
## 1.0.117 (2020-06-04)
#### Others
* mise en place du changelog auto-généré
Full set of changes: [`1.0.116...1.0.117`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.116...1.0.117)
## 1.0.116 (2020-06-03)
#### Fixes
* (args): pour le type bool correction de la valeur par défaut de la valeur par defaut
Full set of changes: [`1.0.115...1.0.116`](https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/1.0.115...1.0.116)
## 1.0.115 (2020-05-29)
#### New Features
* add bumpversion
#### Fixes
* ajout du push increment version

+ 1
- 1
CONTRIBUTING.md View File

@ -2,7 +2,7 @@
```sh
python3 -m venv socle-infra-venv
source socle-infra-venv/bin/activate
source scrippy-remote/bin/activate
pip install -e .
```


+ 3
- 2
Jenkinsfile View File

@ -39,7 +39,7 @@ pipeline {
withPythonEnv('python3') {
sh "pip install --upgrade pip"
sh "python -m pip install .[dev]"
sh "python3 -m pytest --cov=scrippy_remote --junitxml=./junit_report.xml"
}
}
}
@ -71,7 +71,7 @@ pipeline {
withPythonEnv('python3') {
sh "bump2version patch"
String version = getAppVersion()
sh "auto-changelog -v ${version} --issue-url https://gitlab-infra.ref.gnc/python-infra/python-infra/issues/{id} --starting-commit dbd7aa41a423126f5ede4e7c1c8436ca02ce022a -d 'Fichier généré automatiquement par auto-changelog' --diff-url https://gitlab-infra.ref.gnc/python-infra/python-infra/compare/{previous}...{current}"
sh "auto-changelog -v ${version} --issue-url https://gitlab-infra.ref.gnc/scrippy/scrippy-remote/issues/{id} --starting-commit 424aec4d3db56e964595f7c0c5b2bbb7145632cc -d 'Fichier généré automatiquement par auto-changelog' --diff-url https://gitlab-infra.ref.gnc/scrippy/scrippy-remote/compare/{previous}...{current}"
sh "git add setup.cfg .bumpversion.cfg CHANGELOG.md"
sh "git commit -m 'increment version and update changelog [ci-skip]'"
sh "git tag ${version}"
@ -94,6 +94,7 @@ pipeline {
post {
always {
junit allowEmptyResults: true, testResults: 'junit_report.xml'
commentMergeRequest()
cleanWs()
}


+ 17
- 8
Makefile View File

@ -1,7 +1,7 @@
VERSION=$(shell python3 ./setup.py --version)
.RECIPEPREFIX +=
.PHONY: install uninstall all distclean bumpversion autochangelog tag dist push release
.PHONY: install uninstall all distclean test bumpversion autochangelog tag dist push release
all: | uninstall install distclean
@ -9,10 +9,19 @@ install:
sudo python3 ./setup.py build install
uninstall:
sudo pip3 uninstall -y python-infra
sudo pip3 uninstall -y scrippy-remote
distclean:
sudo rm -Rf ./build ./dist ./python_infra.egg-info
sudo rm -Rf ./build \
./dist \
./scrippy_remote.egg-info \
./scrippy_remote/__pycache__ \
./tests/__pycache__ \
./junit_report.xml \
./.coverage
test:
python3 -m pytest --cov=scrippy_remote --junitxml=./junit_report.xml
bumpversion:
bump2version patch
@ -20,10 +29,10 @@ bumpversion:
autochangelog:
VERSION=$(shell python3 ./setup.py --version)
auto-changelog -v "${VERSION}" \
--issue-url https://git.mcos.nc/issues/{id} \
--starting-commit dbd7aa41a423126f5ede4e7c1c8436ca02ce022a \
-d 'Fichier généré automatiquement par auto-changelog' \
--diff-url https://git.mcos.nc/python/infra/src/tag/{current}
--issue-url https://git.mcos.nc/scrippy/scrippy-remote/issues/{id} \
--starting-commit 424aec4d3db56e964595f7c0c5b2bbb7145632cc \
-d 'Fichier généré automatiquement par auto-changelog' \
--diff-url https://git.mcos.nc/scrippy/scrippy-remote/src/tag/{current}
tag:
VERSION=$(shell python3 ./setup.py --version)
@ -38,4 +47,4 @@ push:
git push origin
git push --tags
release: | distclean bumpversion autochangelog tag dist push
release: | distclean build install test bumpversion autochangelog tag dist push

+ 95
- 1562
README.md
File diff suppressed because it is too large
View File


+ 0
- 18
requirements.txt View File

@ -1,19 +1 @@
GitPython>=2.1.15
psycopg2-binary>=2.8.3
markupsafe>=1.1.1
pycparser>=2.19
paramiko>=2.6.0
argcomplete>=1.10.0
bcrypt>=3.1.7
coloredlogs>=10.0
pytz>=2019.3
pynacl>=1.3.0
cryptography>=2.7
six>=1.12.0
jinja2>=2.10.3
asn1crypto>=1.0.1
prettytable>=0.7.2
cffi>=1.12.3
humanfriendly>=4.18
sqlparse>=0.3.0
filelock>=3.0.12

BIN
scrippy-remote.png View File

Before After
Width: 532  |  Height: 334  |  Size: 19 KiB

+ 7
- 0
scrippy_remote/__init__.py View File

@ -0,0 +1,7 @@
class ScrippyRemoteError(Exception):
"""Classe d'erreur spécifique."""
def __init__(self, message):
"""Initialise l'instance."""
self.message = message
super().__init__(self.message)

remote/__init__.py → scrippy_remote/remote/__init__.py View File

@ -1,7 +1,6 @@
#!/bin/env python3
"""
Ce module offre l'ensemble des objets, méthodes et fonctions permettant les opérations sur les hotes distants accessibles via SSH/SFTP:
Ce module offre l'ensemble des objets, méthodes et fonctions permettant les opérations sur les hotes distants accessibles via SSH/SFTP et FTP.
- Exécution de comande sur hôte distant
- Copie de répertoires/fichiers sur hôte distant
- Suppression de répertoires/fichiers sur hôte distant
@ -9,5 +8,5 @@ Ce module offre l'ensemble des objets, méthodes et fonctions permettant les op
- ...
"""
from infra.remote.ssh import Ssh
from infra.remote.ftp import Ftp
from scrippy_remote.remote.ssh import Ssh
from scrippy_remote.remote.ftp import Ftp

remote/ftp.py → scrippy_remote/remote/ftp.py View File

@ -1,15 +1,16 @@
"""Le module scrippy_remote.remote.ftp implémente la partie cliente du protocole FTP sous forme de la classe Ftp."""
import os
import re
import sys
import logging
from infra.remote.gncftp import GncFtp
from scrippy_remote.remote.gncftp import GncFtp
from scrippy_remote import ScrippyRemoteError
class Ftp:
"""
La classe principale permettant la manipulation des hotes distants via FTP
"""
"""La classe principale permettant la manipulation des hotes distants via FTP."""
def __init__(self, hostname, port=21, username="anonymous", password="", tls=True, explicit=True):
"""Initialise le client FTP."""
logging.info("[+] Creation de la connexion:")
self.hostname = hostname
self.port = port
@ -25,17 +26,17 @@ class Ftp:
self.explicit)
def __enter__(self):
"""Point d'entrée."""
self.connect()
return self
def __exit__(self, type_err, value, traceback):
"""Point de sortie."""
del type_err, value, traceback
self.close()
def connect(self, timeout=30):
"""
Permet la connexion FTP a un hôte distant.
"""
"""Se connecte au serveur FTP distant."""
connected = False
logging.info("[+] Connexion a {}@{}:{}".format(self.username, self.hostname, self.port))
try:
@ -50,7 +51,7 @@ class Ftp:
return connected
def close(self):
""" Ferme la connexion passee en argument. """
"""Ferme la connexion."""
logging.info("[+] Fermeture de la connexion a {}@{}".format(self.username, self.hostname))
if self.remote:
self.remote.close()
@ -58,6 +59,7 @@ class Ftp:
def get_file(self, remote_file, local_filepath, create_dir=False):
"""
Recupere le fichier distant 'filepath'.
Si create_dir est positionne a True et si le fichier 'filepath' est un chemin dote de plusieurs composantes (ex: path/to/file), alors l'arbroescence est recree localement dans le repertoire 'local_filepath'.
"""
local_fname = os.path.join(local_filepath, os.path.basename(remote_file))
@ -93,12 +95,14 @@ class Ftp:
logging.debug(" '-> Recuperation des donnees")
self.remote.retrbinary("RETR {}".format(remote_file), open(local_fname, 'wb').write)
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
def put_file(self, filepath, remote_dir='', create_dir=False):
"""
Depose le fichier local 'filepath' sur le serveur distant dans le repertoire 'remote_dir'.
Si 'create_dir' est positionnee a True alors l'arborescence sera recree
sur le serveur distant.
"""
@ -133,12 +137,14 @@ class Ftp:
logging.debug(" '-> Envoi des donnees")
self.remote.storbinary('STOR {}'.format(remote_fname), open(filepath, 'rb'))
except Exception as e:
logging.critical("Erreur lors du transfert: [{}]: {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "Erreur lors du transfert: [{}]: {}".format(e.__class__.__name__, e)
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
def create_local_dirs(self, remote_file, local_filepath):
"""
Creation de l'arborescence de 'remote_file' dans le repertoire local_filepath.
La derniere composante de 'remote_file' est consideree comme un fichier.
"""
hierarchy = remote_file.split('/')[:-1]
@ -150,12 +156,14 @@ class Ftp:
if not os.path.isdir(local_file_path):
os.mkdir(local_file_path)
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
def create_remote_dirs(self, remote_file, remote_dir=''):
"""
Creation de l'arborescence de 'remote_file' sur l'hote distant a partir de 'remote_dir'.
La derniere composante de 'remote_file' est consideree comme un fichier.
"""
hierarchy = remote_file.split('/')[:-1]
@ -166,32 +174,32 @@ class Ftp:
try:
self.remote.mkd(remote_dir)
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
def delete_remote_file(self, remote_file):
"""
Supprime le fichier distant dont le chemin complet est passe en argument.
"""
"""Supprime le fichier distant dont le chemin complet est passe en argument."""
try:
self.remote.delete(remote_file)
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
def delete_remote_dir(self, remote_dir):
"""
Supprime le repertoire distant dont le chemin complet est passe en argument.
"""
"""Supprime le repertoire distant dont le chemin complet est passe en argument."""
try:
self.remote.rmd(remote_dir)
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
def list(self, remote_dir, file_type='f', pattern='.*'):
"""
Renvoie la liste des fichiers presents dans le repertoire remote_dir.
L'argument file_type permet de selectionner le type de fichier liste (f=file (valeur par defaut), d=directory).
"""
content = []
@ -204,8 +212,8 @@ class Ftp:
reg = re.compile("^d.*")
content = [os.path.join(remote_dir, f.split()[-1]) for f in content if re.match(reg, f)]
reg = re.compile(pattern)
content = [os.path.join(remote_dir, f.split()[-1]) for f in content if re.match(reg, f)]
return content
return [os.path.join(remote_dir, f.split()[-1]) for f in content if re.match(reg, f)]
except Exception as e:
logging.critical(" '-> [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)

remote/gncftp.py → scrippy_remote/remote/gncftp.py View File

@ -4,11 +4,10 @@ ainsi que le FTPS sur n'importe quel port TCP.
© 2020 - MCO System - https://www.mcos.nc
"""
import logging
import re
import socket
import ssl
import socket
import logging
from socket import _GLOBAL_DEFAULT_TIMEOUT
# ------------------------------------------------------------------------------

remote/ssh.py → scrippy_remote/remote/ssh.py View File

@ -1,16 +1,18 @@
"""Le module scrippy_remote.remote.ssh implémente la partie cliente du protocole SSH/SFTP sous forme de la classe Ssh."""
import os
import re
import sys
import stat
import socket
from time import sleep
import paramiko
import logging
import paramiko
from time import sleep
from scrippy_remote import ScrippyRemoteError
def clean_path(path):
"""
On supprime les éventuels / finaux de path
Supprime les éventuels / finaux de path.
:param path: path à nettoyer
:return: renvoie path sans le '/' final si présents
"""
@ -29,11 +31,12 @@ class Ssh:
La classe principale permettant la manipulation des hôtes distants via SSH.
Cette classe permet
- L'exécution de commande distante
- L'exécution de commandes distantes
- Le transfert de fichier
"""
def __init__(self, username, hostname, port, password=None, key_filename=None):
"""Initialise le client SSH."""
if not logging.getLogger().level == logging.DEBUG:
logging.getLogger("paramiko").setLevel(logging.ERROR)
@ -45,19 +48,20 @@ class Ssh:
self.remote = None
def __enter__(self):
"""Point d'entrée."""
self.connect()
return self
def __exit__(self, type_err, value, traceback):
"""Point de sortie."""
del type_err, value, traceback
self.close()
def connect(self):
"""
Permet la connexion à un autre distant.
Se connecte à un hôte distant.
Le chemin vers une clef publique SSH alternative peut être passée à la fonction via l'option key_filename.
Dans tous les cas le répertoire ~/.ssh de l'utilisateur courant est parcouru afin de trouver le clef SSH adaptée.
Une erreur est levée et False retourné dans au moins les cas suivants:
@ -93,21 +97,25 @@ class Ssh:
port=self.port,
username=self.username)
except paramiko.BadHostKeyException as e:
logging.critical(" '-> La clef de l'hote distant n'est pas celle attendue: [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "La clef de l'hote distant n'est pas celle attendue: [{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
except paramiko.AuthenticationException as e:
logging.critical(" '-> Echec de l'authentification: [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "Echec de l'authentification: [{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
except (paramiko.SSHException,
socket.gaierror,
paramiko.ssh_exception.NoValidConnectionsError,
FileNotFoundError) as e:
logging.critical(" '-> Echec de la connexion: [{}] {}".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "Echec de la connexion: [{}] {}".format(e.__class__.__name__, e)
logging.critical(" '-> {}".format(err_msg))
raise ScrippyRemoteError(err_msg)
except Exception as e:
logging.critical(" +-> Erreur inattendue: [{}] {}".format(e.__class__.__name__, e))
err_msg = "Erreur inattendue: [{}] {}".format(e.__class__.__name__, e)
logging.critical(" +-> {}".format(err_msg))
logging.critical(" '-> Merci de reporter l'erreur ci-dessus afin qu'elle soit integree au module infra.")
sys.exit(1)
raise ScrippyRemoteError(err_msg)
def close(self):
""" Ferme la connexion passée en argument. """
@ -170,10 +178,10 @@ class Ssh:
if return_stdout:
res['stdout'] = stdout_content
return res
except paramiko.SSHException as e:
logging.error("Erreur lors de l'execution de la commande: [{}] {}: ".format(e.__class__.__name__, e))
sys.exit(1)
err_msg = "Erreur lors de l'execution de la commande: [{}] {}: ".format(e.__class__.__name__, e)
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
# -- PUT -----------------------------------------------------------------------
@ -257,8 +265,9 @@ class Ssh:
err += 1
logging.warning(" '-> [{}] {}".format(e.__class__.__name__, e))
if exit_on_error:
logging.critical("Erreur lors du transfert et exit_on_error positionnee a True: Arret immediat.")
sys.exit(1)
err_msg = "Erreur lors du transfert et exit_on_error positionnee a True: Arret immediat."
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
return err
def delete_local_files(self, local_files, exit_on_error):
@ -274,8 +283,9 @@ class Ssh:
err += 1
logging.warning(" '-> [{}] {}".format(e.__class__.__name__, e))
if exit_on_error:
logging.critical("Erreur lors de la suppression et exit_on_error positionnee a True: Arret immediat.")
sys.exit(1)
err_msg = "Erreur lors de la suppression et exit_on_error positionnee a True: Arret immediat."
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
return err
# -- GET -----------------------------------------------------------------------
@ -338,10 +348,12 @@ class Ssh:
for directory in remote_dirs:
remote_files += self.find_remote_files(directory, pattern, recursive, exit_on_error)
except Exception as e:
logging.warning("Erreur lors de recuperation: [{}] {}".format(e.__class__.__name__, str(e)))
err_msg = "Erreur lors de recuperation: [{}] {}".format(e.__class__.__name__, str(e))
logging.warning(err_msg)
if exit_on_error:
logging.critical("Erreur lors de la recuperation de la liste des fichiers et exit_on_error positionnee a True: Arret immediat.")
sys.exit(1)
err_msg = "Erreur lors de la recuperation de la liste des fichiers et exit_on_error positionnee a True: Arret immediat."
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
return remote_files
def transfer_remote_files(self, local_path, remote_files, exit_on_error):
@ -359,10 +371,12 @@ class Ssh:
sftp.get(remote_file, local_fname)
except Exception as e:
err += 1
logging.warning("[{}] {}".format(e.__class__.__name__, e))
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.warning(err_msg)
if exit_on_error:
logging.critical("Erreur lors du transfert et exit_on_error positionnee a True: Arret immediat.")
sys.exit(1)
err_msg = "Erreur lors du transfert et exit_on_error positionnee a True: Arret immediat."
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
return err
def delete_remote_files(self, remote_files, exit_on_error):
@ -377,10 +391,12 @@ class Ssh:
sftp.remove(remote_file)
except Exception as e:
err += 1
logging.warning("[{}] {}".format(e.__class__.__name__, e))
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.warning(err_msg)
if exit_on_error:
logging.critical("Erreur lors de la suppression et exit_on_error positionnee a True: Arret immediat.")
sys.exit(1)
err_msg = "Erreur lors de la suppression et exit_on_error positionnee a True: Arret immediat."
logging.critical(err_msg)
raise ScrippyRemoteError(err_msg)
return err
# -- DELETE --------------------------------------------------------------------
@ -424,6 +440,7 @@ class Ssh:
sftp.mkdir(basename)
sftp.chdir(basename)
except Exception as e:
logging.error("[{}] {}".format(e.__class__.__name__, e))
raise e
err_msg = "[{}] {}".format(e.__class__.__name__, e)
logging.error(err_msg)
raise ScrippyRemoteError(err_msg)
return True

+ 4
- 20
setup.cfg View File

@ -4,9 +4,9 @@ classifiers =
Programming Language :: Python :: 3
License :: WTFPL
Operating System :: POSIX :: Linux
url = https://gitlab-infra.ref.gnc/puppet/python-infra
url = https://gitlab-infra.ref.gnc/scrippy/scrippy-remote
description = "Socle de fonctions utilisées dans les scripts python de gestion d'infrastructure."
name = python-infra
name = scrippy-remote
long_description = file: README.md
long_description_content_type = text/markdown
author = Michael Costa, Florent Chevalier
@ -15,25 +15,7 @@ author_email = michael.costa@mcos.nc, florent.chevalier@gouv.nc
[options]
packages = find:
install_requires =
GitPython>=2.1.15
psycopg2-binary>=2.8.3
markupsafe>=1.1.1
pycparser>=2.19
paramiko>=2.6.0
argcomplete>=1.10.0
bcrypt>=3.1.7
coloredlogs>=10.0
pytz>=2019.3
pynacl>=1.3.0
cryptography>=2.7
six>=1.12.0
jinja2>=2.10.3
asn1crypto>=1.0.1
prettytable>=0.7.2
cffi>=1.12.3
humanfriendly>=4.18
sqlparse>=0.3.0
filelock>=3.0.12
[options.extras_require]
dev =
@ -49,8 +31,10 @@ dev =
pep8-naming>=0.8.2
flake8-tabs==2.1.0
pytest==5.2.1
pytest-cov>=2.8.1
bump2version==0.5.11
auto-changelog==0.5.1
setuptools>=39.2.0
[flake8]
ignore = E111


+ 2
- 2
sonar-project.properties View File

@ -1,5 +1,5 @@
sonar.projectName=python-infra
sonar.gitlab.project_id=puppet/python-infra
sonar.projectName=scrippy-mail
sonar.gitlab.project_id=scrippy/scrippy-mail
sonar.gitlab.url=https://gitlab-infra.ref.gnc


+ 7
- 0
tests/__init__.py View File

@ -0,0 +1,7 @@
"""Test scrippy_mail."""
from scrippy_remote import remote
def test_remote():
"""Test remote."""
assert True

+ 50
- 0
tests/test_remote_ssh.py View File

@ -0,0 +1,50 @@
"""Module de test scrippy_remote.remote.Ssh."""
import os
import logging
from scrippy_remote.remote import Ssh, ScrippyRemoteError
def test_remote_cmd():
"""Test d'exécution de commande."""
remote_host = "srv-trinity-01.preprod-ref.gnc"
remote_port = 22
remote_user = "jenkins"
key_filename = "/var/lib/jenkins/.ssh/id_rsa"
with Ssh(username=remote_user,
hostname=remote_host,
port=remote_port,
key_filename=key_filename) as host:
stdout = host.exec_command("ls /", return_stdout=True)
assert stdout["exit_code"] == 0
def test_get_file():
"""Test de récuperation de fichier."""
remote_host = "srv-trinity-01.preprod-ref.gnc"
remote_port = 22
remote_user = "jenkins"
key_filename = "/var/lib/jenkins/.ssh/id_rsa"
remote_path = "/var/lib/jenkins"
local_path = "/var/lib/jenkins/.ssh/known_hosts"
test_filename = "jenkins.CLI.xml"
pattern = ".*{}".format(test_filename)
recursive = False
delete = False
exit_on_error = True
with Ssh(username=remote_user,
hostname=remote_host,
port=remote_port,
key_filename=key_filename) as host:
try:
err = host.sftp_get(remote_path=remote_path,
local_path=local_path,
pattern=pattern,
recursive=recursive,
delete=delete,
exit_on_error=exit_on_error)
logging.info("Errors: {}".format(err))
assert os.path.isfile(os.path.join(local_path, test_filename))
except ScrippyRemoteError as e:
logging.critical("{}".format(e))

Loading…
Cancel
Save